Testing
Overview
Section titled “Overview”Hard rule 4: a feature is done when it works on desktop and in melonDS. Logic is deterministic (see determinism), so “works on both” can be checked mechanically: play the same input log on both platforms and compare what they print and what they draw.
Tests are written first, and logic crates are tested on the host. Most bugs should be
caught by cargo test in seconds, not in an emulator.
Layer 1: host tests
Section titled “Layer 1: host tests”cargo test runs every workspace crate’s unit, integration and doc tests on the
host. The logic crates are no_std but compile for the host, and each has a fake
for what it cannot reach:
| Crate | Tested against |
|---|---|
pl-battle |
an in-memory Dex and a forced RNG; ported Showdown scenarios keep their attribution |
pl-script |
pl-scriptc’s fake.rs, a stand-in host that records every call |
pl-world, pl-ui, pl-systems |
their own small fixtures |
pl-game |
fake.rs, a fake platform; whole sessions replayed from input logs |
pl-import |
tiny synthetic files built in code; a real-ROM test is ignored by default |
cargo xtask test (xtask/src/test_host.rs) runs the same tests but skips a crate
whose tests passed with nothing they depend on changed since: the toolchain, the
crate’s files, the workspace crates it depends on, and data files it names. Failures
are never cached. --no-cache runs everything.
Some host tests guard the repository itself (xtask/tests/):
docs.rs: the numbers indocs/registry.md(hooks, save sections, game flags, format versions) match the code; decisions are numbered once and in order; every task file is listed; plan numbers are used once.present.rs: no logic crate can reachpl-present, directly or through another crate.
Input logs, digests and the replay proof
Section titled “Input logs, digests and the replay proof”pl-game records the input of every tick and replays it: a 4-byte record a tick, the
clock at tick 0 in the header, and records for clock moves and network bytes, so even
an online session replays with no network (crates/pl-game/src/input.rs). A replayed session is
the same session on any platform. crates/pl-game/demo-replay/ holds recorded
sessions the tests play.
After a tick the engine can report digests (Digests): World::state_hash, the
scripts’ store, machine and generators, and Battle::state_hash. Each is FNV-1a over
a fixed byte encoding of every field, so it does not depend on word size or
addresses. present_tests.rs replays a session at two presentation presets and
requires every tick’s digests, the log, the sounds, the saves and the re-recorded
input log to be equal; a planted leak must make it fail.
Layer 2: golden images
Section titled “Layer 2: golden images”crates/pl-desktop/tests/golden.rs runs the real desktop binary headless on fixed
input logs and compares its screenshots with the PNGs in
crates/pl-desktop/tests/golden/, pixel for pixel. The renderer is integer only,
so the pictures are equal on every system. The wide profile has its own pictures,
which are its only check, since the DS cannot show it.
Layer 3: cargo xtask test-nds
Section titled “Layer 3: cargo xtask test-nds”test-nds plays each session on the desktop (headless) and in melonDS (on a hidden,
silent display), and compares the two (xtask/src/test_nds.rs, headless.rs,
nds_headless.py).
Sessions are rows of xtask/src/sessions.rs. A row says what is played and on
which ROM, how the screens are compared, how long the DS may take, where it saves,
which session’s save it continues, and the log lines it must and must not have.
Adding a session is adding a row. A test refuses a name used twice.
What must agree:
- The logs. The field prints
pl-field:lines (warps, scripts, battles, astate=hash); battles, the AI duels and link battles print theirs. Desktop and DS must print the same lines for the same input. - The pictures. Mode
2dcompares every pixel and requires them equal. Mode3dcompares the top screen loosely: at most 1 % of pixels may differ by more than 16/255.3d@ROWScompares a 2D window over the 3D scene exactly. Colours are compared as the DS’s 15-bit colours, because the two widen 5-bit channels to 8 bits differently.
Running it. cargo xtask test-nds NAME... runs only those sessions and the ones
they continue from; --list names them; a failed session does not stop the rest.
-j N runs sessions in parallel. melonDS runs on its interpreter by default for
fidelity; PL_NDS_JIT=1 uses its JIT for a quick run. PL_NDS_TIMEOUT_SCALE=2
helps on a busy machine. test-nds --project DIR --replay FILE plays a project’s
own game on both platforms.
Caching. A session’s key covers what can change how it ends: the ROM’s bytes, the
desktop program and what it reads, melonDS, the scripts that play and judge it, its
row, and the keys of the sessions whose saves it continues. A pass is kept with its
pictures; a failure never is (xtask/src/nds_cache.rs).
Test programs (the 3D test scene, the interface gallery, the sound checks) are in the
test ROMs only, behind pl-nds’s test-programs feature, never in a game’s ROM.
Related commands
Section titled “Related commands”| Command | Checks |
|---|---|
cargo xtask test-sound |
the sound check on both platforms, measured from captured audio (nothing is played aloud) |
cargo xtask test-part1 |
the demo’s part 1 from a new game to the end, as one session |
cargo xtask link [dd|dsd|dsds] |
a link battle on each pairing of platforms through an in-process server; both sides’ battle digests must match |
cargo xtask profile-nds |
heap, time and stack against the budgets (see budgets) |
cargo xtask size |
code size against its budget |
Blessing
Section titled “Blessing”Several files the tests compare are generated: the sound of the sample and demo, the
game.toml key reference and its siblings, the sample’s generated project files, the
packs, the input logs and the demo’s start save, the golden images, and the script
language reference. When a change is meant to alter them, cargo xtask bless
(xtask/src/bless.rs) rewrites them all in order, because each depends on the
one before:
- sound and the settings references (
PL_BLESS=1 cargo test -p pl-assets); - the sample’s generated files (
PL_BLESS=1 cargo test -p pl-samplekit); - the packs of both games (built, not checked in);
- input logs and the demo’s start save (
PL_BLESS=1 cargo test -p pl-game); - golden images (
PL_BLESS=1 cargo test -p pl-desktop); - the script language reference (
PL_BLESS=1 cargo test -p pl-scriptc --test reference).
It prints which files changed. Look at every changed picture before keeping it: a golden image that changed is a picture that changed, and blessing a wrong one hides the bug it would have caught.
Showdown fuzzer
Section titled “Showdown fuzzer”Built-in battle rules are checked against Pokémon Showdown, not memory (hard rule 5;
see Showdown conformance). The harness is
conformance/, a development-only crate that needs Node:
conformance/showdown.jsgenerates battles (random teams and legal choices from a stage’s list of moves, abilities and items) and plays them in Showdown, recording the protocol lines and every random number drawn.conformance/src/lib.rsreplays each inpl-battleand compares the traces line by line.
A new effect joins the newest stage or a new stage at the end, never an older one, because that would change what every later case generates. A batch runs until a few thousand battles of its stage and the earlier ones match, then 20,000 before it is kept. A mismatch is never acceptable “for now”. Showdown’s data is read at run time on the developer’s machine; nothing of it is stored.
A game’s own mechanics are kept apart: the engine’s conformance runs never link a
game crate. A game crate tests its mechanics with scenarios that play a battle with
and without its tables (templates/game-crate/tests/scenarios.rs).
- Planned (plan 34): a small text format for creators’ battle scenarios, run by
cargo testandploom test, with baselines;ploom test --officialproves a build changes nothing it did not declare.
Fidelity pairs
Section titled “Fidelity pairs”For imported games, cargo xtask pairs <game> puts the real game (in melonDS, from a
save, at a set clock) beside the engine’s import at the same map, cell, facing, hour
and story state, and counts the difference. Each game is a profile in xtask/pairs/.
A 3D view passes when at most 3 % of pixels differ by more than 16/255; a 2D screen
at 99 % equal. The references come from the user’s own copy and stay private. See
fidelity pairs.
.github/workflows/ci.yml runs format, clippy, host tests and the DS build’s size
check, inside the project’s development shell. It runs no emulator and no Showdown
conformance (both need things outside the repository), and leaves out Windows and
macOS builds on purpose until someone reads their result. Its first run on GitHub has
not been observed.
Checklist
Section titled “Checklist”cargo fmt --all --checkcargo clippy --all-targets -- -D warningscargo xtask testcargo xtask sizecargo xtask test-ndsplus clippy for the DS target, the fuzzer for a battle change, and pairs for a change meant to bring an import closer to its original. Say plainly what was not run.