175. Round-trip tests across engines

Two engines, one answer

Chapter 112 compared a program with its compiled version: same behaviour, or a bug. Now there is a second compiler. The Rust engine claims to make the TypeScript engine’s decisions and print its exact output: the same JavaScript or Python, and the same explain text, for every input. “Mostly the same” would be worse than useless. A playground running the wasm build would disagree with the CLI, and nobody could tell which one to believe.

Fixtures from the reference

The veteran’s pages are written once. engine/rust/scripts/gen_expected.mjs imports the TypeScript compiler and runs it over six suites: the JS corpus, the end-to-end benchmarks, and extra programs written for the port, plus the same three for Python. Its core loop, adapted:

const variants = [
  ['', {}],
  ['.assume', { bounds: 'assume' }],
];
for (const [suffix, extra] of variants) {
  const r = compile(src, jsPlugin(), extra);
  write(`${base}${suffix}.js`, r.code);
  write(`${base}${suffix}.explain.txt`,
    r.explain);
}

The .assume variant takes the emitter’s unchecked paths, which the default (bounds checks on) never reaches. The files land in tests/expected and are committed. When the reference changes, someone reruns the script and reviews the diff.

The Rust side

crates/cli/tests/parity.rs is a Cargo integration test. It compiles the same sources with the Rust engine and compares (adapted):

for p in programs() {
    for (suffix, bounds) in VARIANTS {
        let name =
            format!("{}{suffix}", p.name);
        let (code, explain) =
            expected(&p, &name);
        let r = p.compile(bounds);
        if r.code != code { /* record */ }
        if r.explain != explain { /*…*/ }
    }
}

It records every mismatch and fails once, listing them all. Each entry says where the texts first differ, which is your exercise. Other tests in the same file check that the fixture folder matches the programs exactly (none missing, none stale), that the 14 explain --frames fixtures and four --profile fixtures match too, that each corpus program’s // viscosity: ps=soa header holds, and that the built viscosity binary prints the same output. Cargo tells the test where the binary is through CARGO_BIN_EXE_viscosity^[1].

When I ran it (rustc 1.97, on a copy of the repo with its own target directory, so as not to disturb other work):

parity: 190/190 (54 Python)

That is 95 programs, each in two variants. The count grows as programs are added (it was 186 when this chapter was first written). What matters is that the two numbers are equal.

Same bytes isn’t the same as right

Two engines can agree on a wrong answer. So differential.rs runs each original and compiled JavaScript program with node (in an isolated VM context, by a copy of the TS runner) and compares everything they print and throw. Python programs run with python3. On my run, 68 of 68 JavaScript and 27 of 27 Python programs behaved identically.

To see both tests at work, I deleted one line from a copy of crates/core/src/builder.rs: the case that rewrites a throw operand when a trivial phi is removed (chapter 176). Parity, trimmed (this was when it had 186 pairs):

4 of 186 differ:
corpus_throw_from_loop.js: line 1:
  want: // viscosity: ps=aos
  got:  // Generated by Viscosity: "ps" ...

and the differential test:

corpus_throw_from_loop:
  original: {"out":["object 1"],…}
  compiled: {"out":["number undefined"],…}

Parity says something changed in exactly one program. The differential test says why it matters: the compiled program throws a number where the original threw an object.

What parity forces

Matching decisions isn’t enough. explain lists its reasons in the order the analyses find them, and the emitter’s edits come in the order the plan is walked. So the port copies the reference’s order too. A JS Map iterates in insertion order, so where the TS code uses one, the port uses IndexMap. Sorts are stable. Use lists keep duplicates, and the numeric fixpoint updates in the same sequence. Numbers are formatted by vir::js_number, because Rust and JavaScript print some doubles differently:

fn main() {
    // JavaScript's String(x), captured
    // once from Node.
    let x3 = 0.1 + 0.2;
    let fixtures = [
        (x3, "0.30000000000000004"),
        (100.0, "100"),
        (1e21, "1e+21"),
        (1e-7, "1e-7"),
        (-0.0, "0"),
    ];
    let mut bad = 0;
    for (x, want) in fixtures {
        let got = format!("{}", x);
        if got != want {
            bad += 1;
            println!("{want} vs {got}");
        }
    }
    let n = fixtures.len();
    let ok = n - bad;
    println!("{ok} of {n} match");
}

Ten bugs in the reference

When the engines disagree, the obvious suspect is the newcomer. It isn’t always guilty. The README lists ten bugs the port turned up in the TypeScript engine. Each was fixed in both, and each now has a test program:

1. A throw of a removed phi wasn’t seen as an escape (a miscompile).
2. push({ y: (a, b) }) lost its parentheses (chapter 174).
3. Shapes were compared by joined names, so 'a,b' matched a, b.
4. export default function () {} crashed the resolver.
5. Annotation comments split lines at \n only.
6. A number at the very end of a Python file read past the string: 1 became a float, and 0x1 hung the tokenizer.
7. f-string dataclass defaults were copied to each call (a miscompile).
8. global g with no module-level g crashed.
9. len() of an element crashed when a field was named length.
10. A rebound dataclass name was still compiled as the class (a miscompile).

Bug 6 is a nice example of what a second implementation catches. Past the end, JavaScript’s src[i] is undefined, and a regex test turns that into the string "undefined", which contains an e. The Rust tokenizer reads s.get(k), which past the end gives no character at all, so the two engines disagreed.

Your turn

// The message for differing line n.
fn line_msg(n: usize, w: &str, g: &str)
    -> String {
    format!("line {n}: {w:?} vs {g:?}")
}

fn first_difference(
    want: &str,
    got: &str,
) -> Option<String> {
    todo!()
}

What you learned

• A cross-implementation test holds a new engine to a reference: same inputs, byte-identical outputs.
• gen_expected.mjs writes the reference’s fixtures (two variants per program). parity.rs compares the Rust engine with them on every cargo test, and checks for missing or stale fixtures.
• Equal bytes can still be wrong together, so a differential test runs both programs in node and python3.
• Byte parity forces the reference’s order: IndexMap, stable sorts, and JavaScript number formatting.
• The disagreements found ten bugs in the reference, fixed in both engines.