Skip to content

Testing

Hard rule 4: a feature is done when it works on desktop and in melonDS. Logic is deterministic (see determinism), so “works on both” can be checked mechanically: play the same input log on both platforms and compare what they print and what they draw.

Tests are written first, and logic crates are tested on the host. Most bugs should be caught by cargo test in seconds, not in an emulator.

cargo test runs every workspace crate’s unit, integration and doc tests on the host. The logic crates are no_std but compile for the host, and each has a fake for what it cannot reach:

Crate Tested against
pl-battle an in-memory Dex and a forced RNG; ported Showdown scenarios keep their attribution
pl-script pl-scriptc’s fake.rs, a stand-in host that records every call
pl-world, pl-ui, pl-systems their own small fixtures
pl-game fake.rs, a fake platform; whole sessions replayed from input logs
pl-import tiny synthetic files built in code; a real-ROM test is ignored by default

cargo xtask test (xtask/src/test_host.rs) runs the same tests but skips a crate whose tests passed with nothing they depend on changed since: the toolchain, the crate’s files, the workspace crates it depends on, and data files it names. Failures are never cached. --no-cache runs everything.

Some host tests guard the repository itself (xtask/tests/):

  • docs.rs: the numbers in docs/registry.md (hooks, save sections, game flags, format versions) match the code; decisions are numbered once and in order; every task file is listed; plan numbers are used once.
  • present.rs: no logic crate can reach pl-present, directly or through another crate.

pl-game records the input of every tick and replays it: a 4-byte record a tick, the clock at tick 0 in the header, and records for clock moves and network bytes, so even an online session replays with no network (crates/pl-game/src/input.rs). A replayed session is the same session on any platform. crates/pl-game/demo-replay/ holds recorded sessions the tests play.

After a tick the engine can report digests (Digests): World::state_hash, the scripts’ store, machine and generators, and Battle::state_hash. Each is FNV-1a over a fixed byte encoding of every field, so it does not depend on word size or addresses. present_tests.rs replays a session at two presentation presets and requires every tick’s digests, the log, the sounds, the saves and the re-recorded input log to be equal; a planted leak must make it fail.

crates/pl-desktop/tests/golden.rs runs the real desktop binary headless on fixed input logs and compares its screenshots with the PNGs in crates/pl-desktop/tests/golden/, pixel for pixel. The renderer is integer only, so the pictures are equal on every system. The wide profile has its own pictures, which are its only check, since the DS cannot show it.

test-nds plays each session on the desktop (headless) and in melonDS (on a hidden, silent display), and compares the two (xtask/src/test_nds.rs, headless.rs, nds_headless.py).

Sessions are rows of xtask/src/sessions.rs. A row says what is played and on which ROM, how the screens are compared, how long the DS may take, where it saves, which session’s save it continues, and the log lines it must and must not have. Adding a session is adding a row. A test refuses a name used twice.

What must agree:

  • The logs. The field prints pl-field: lines (warps, scripts, battles, a state= hash); battles, the AI duels and link battles print theirs. Desktop and DS must print the same lines for the same input.
  • The pictures. Mode 2d compares every pixel and requires them equal. Mode 3d compares the top screen loosely: at most 1 % of pixels may differ by more than 16/255. 3d@ROWS compares a 2D window over the 3D scene exactly. Colours are compared as the DS’s 15-bit colours, because the two widen 5-bit channels to 8 bits differently.

Running it. cargo xtask test-nds NAME... runs only those sessions and the ones they continue from; --list names them; a failed session does not stop the rest. -j N runs sessions in parallel. melonDS runs on its interpreter by default for fidelity; PL_NDS_JIT=1 uses its JIT for a quick run. PL_NDS_TIMEOUT_SCALE=2 helps on a busy machine. test-nds --project DIR --replay FILE plays a project’s own game on both platforms.

Caching. A session’s key covers what can change how it ends: the ROM’s bytes, the desktop program and what it reads, melonDS, the scripts that play and judge it, its row, and the keys of the sessions whose saves it continues. A pass is kept with its pictures; a failure never is (xtask/src/nds_cache.rs).

Test programs (the 3D test scene, the interface gallery, the sound checks) are in the test ROMs only, behind pl-nds’s test-programs feature, never in a game’s ROM.

Command Checks
cargo xtask test-sound the sound check on both platforms, measured from captured audio (nothing is played aloud)
cargo xtask test-part1 the demo’s part 1 from a new game to the end, as one session
cargo xtask link [dd|dsd|dsds] a link battle on each pairing of platforms through an in-process server; both sides’ battle digests must match
cargo xtask profile-nds heap, time and stack against the budgets (see budgets)
cargo xtask size code size against its budget

Several files the tests compare are generated: the sound of the sample and demo, the game.toml key reference and its siblings, the sample’s generated project files, the packs, the input logs and the demo’s start save, the golden images, and the script language reference. When a change is meant to alter them, cargo xtask bless (xtask/src/bless.rs) rewrites them all in order, because each depends on the one before:

  1. sound and the settings references (PL_BLESS=1 cargo test -p pl-assets);
  2. the sample’s generated files (PL_BLESS=1 cargo test -p pl-samplekit);
  3. the packs of both games (built, not checked in);
  4. input logs and the demo’s start save (PL_BLESS=1 cargo test -p pl-game);
  5. golden images (PL_BLESS=1 cargo test -p pl-desktop);
  6. the script language reference (PL_BLESS=1 cargo test -p pl-scriptc --test reference).

It prints which files changed. Look at every changed picture before keeping it: a golden image that changed is a picture that changed, and blessing a wrong one hides the bug it would have caught.

Built-in battle rules are checked against Pokémon Showdown, not memory (hard rule 5; see Showdown conformance). The harness is conformance/, a development-only crate that needs Node:

  • conformance/showdown.js generates battles (random teams and legal choices from a stage’s list of moves, abilities and items) and plays them in Showdown, recording the protocol lines and every random number drawn.
  • conformance/src/lib.rs replays each in pl-battle and compares the traces line by line.

A new effect joins the newest stage or a new stage at the end, never an older one, because that would change what every later case generates. A batch runs until a few thousand battles of its stage and the earlier ones match, then 20,000 before it is kept. A mismatch is never acceptable “for now”. Showdown’s data is read at run time on the developer’s machine; nothing of it is stored.

A game’s own mechanics are kept apart: the engine’s conformance runs never link a game crate. A game crate tests its mechanics with scenarios that play a battle with and without its tables (templates/game-crate/tests/scenarios.rs).

  • Planned (plan 34): a small text format for creators’ battle scenarios, run by cargo test and ploom test, with baselines; ploom test --official proves a build changes nothing it did not declare.

For imported games, cargo xtask pairs <game> puts the real game (in melonDS, from a save, at a set clock) beside the engine’s import at the same map, cell, facing, hour and story state, and counts the difference. Each game is a profile in xtask/pairs/. A 3D view passes when at most 3 % of pixels differ by more than 16/255; a 2D screen at 99 % equal. The references come from the user’s own copy and stay private. See fidelity pairs.

.github/workflows/ci.yml runs format, clippy, host tests and the DS build’s size check, inside the project’s development shell. It runs no emulator and no Showdown conformance (both need things outside the repository), and leaves out Windows and macOS builds on purpose until someone reads their result. Its first run on GitHub has not been observed.

Terminal window
cargo fmt --all --check
cargo clippy --all-targets -- -D warnings
cargo xtask test
cargo xtask size
cargo xtask test-nds

plus clippy for the DS target, the fuzzer for a battle change, and pairs for a change meant to bring an import closer to its original. Say plainly what was not run.