164. Testing against the shared corpus

One folder, three interpreters

book/companion/corpus holds 265 .drip files, in 30 folders and three loose files. Here is one, super/extra_arguments.drip, with its two comments taken off the ends of lines 9 and 10:

class Base {
  foo(a, b) {
    print "Base.foo(" + a + ", " + b + ")";
  }
}

class Derived < Base {
  foo() {
    print "Derived.foo()";
    super.foo("a", "b", "c", "d");
  }
}

Derived().foo();

Line 9 ends with // expect: Derived.foo(), and line 10 with // expect runtime error: Expected 2 arguments but got 4.

The comments are the test. Three suites read the same files: the tree-walker’s and the TypeScript VM’s, under vitest, and drip-rs’s tests/corpus.rs. None of them has its own copy of the programs or of the answers.

The programs are adapted from the test suite of Crafting Interpreters, which is MIT-licensed, Copyright (c) 2015 Robert Nystrom^[1]. The licence sits beside them in LICENSE-craftinginterpreters, and the folder’s README says what changed: fun became fn, var became let, and identifiers that would clash with the new keywords were renamed. npm run corpus:import in book/companion/ts regenerates them.

Answers in the margin

Five kinds of comment carry expectations:

// expect: <one line of stdout>
// expect runtime error: <message>
// Error ...            (on this line)
// [line N] Error ...   (on line N)
// [c line N] Error ... / [java line N]

The last kind exists because the two interpreters in Crafting Interpreters report compile errors differently. unexpected_character .drip starts:

// [line 3] Error: Unexpected character.
// [java line 3] Error at 'b': Expect ')'
//   after arguments.
foo(a | b);

The VM is clox’s twin, so c expectations apply to it and java ones don’t. The tree-walker is jlox’s twin and takes the java ones, which is chapter 163’s difference: it scans the whole file first and reports more errors. A // nontest comment would turn a file off; none of the 265 has one. All together the corpus has 504 // expect: lines, in 133 files.

A runner without regexes

The TypeScript runner’s check() matches each comment with a regular expression. drip-rs has no dependencies, so tests/common/mod.rs, a port of check(), matches each by hand and keeps the regex in the doc comment:

/// `// expect: ?(.*)`: the text after
/// `// expect:` and one optional space.
fn expect_output(line: &str)
    -> Option<&str>
{
    let rest = after(line, "// expect:")?;
    Some(rest.strip_prefix(' ')
        .unwrap_or(rest))
}

after returns the rest of the line after a marker, cut at any \r, because JavaScript’s . doesn’t match \r and the port must agree with it on every file. Then check compares, in order:

• a runtime error: stderr[0] must be the message, and the first [line N] in the trace must be the comment’s line;
• otherwise every stderr line must be an expected compile error, and every expected one must appear;
• the exit code: 0, 65 or 70;
• stdout, line by line.

tests/common/mod.rs is shared by two test files, each of which says mod common;. Cargo compiles every file in tests/ as its own crate, and a module in a subfolder is the documented way to share code between them^[2]. Each crate uses a different part of it, so the module starts with #![allow(dead_code)].

Its own main

Cargo.toml turns the usual test harness off for both corpus suites:

[[test]]
name = "corpus"
harness = false

With harness = false the file must define main() itself^[2]. tests/corpus.rs’s main is two lines: build the Options, and call run_suite. That buys one line per program, like the TypeScript runner, where libtest would have seen a single #[test] function. Arguments that don’t start with - filter by substring:

$ cargo test --test corpus -- super/
test super/bound_method.drip ... ok
...
corpus result: ok. 20 passed; 0 failed

The cost: libtest catches each test’s panic so that it fails only that test, and a hand-written main has to do that itself. The first version of this runner didn’t. With chapter 163’s broken copy of drip-rs it printed eight ok lines, the ninth program, assignment/undefined.drip, panicked, and the other 237 never ran. So the runner now wraps each program in catch_unwind:

let run = std::panic::catch_unwind(
    AssertUnwindSafe(|| {
        drip::run(&src, opts)
    }),
);

An Err becomes a failure that names the program and quotes the panic message (panicked: use after free: …), and the loop moves on.

What runs, and how hard

SKIP_VM is ["scanning", "expressions", "benchmark"], exactly the TypeScript suite’s SKIP.vm: the first two hold early-chapter tests that aren’t whole programs, and the benchmarks are slow. 265 − 6 − 2 − 11 leaves 246. The tree-walker’s list skips six limit/ programs too, because jlox has none of clox’s limits.

The 246 run with debug_gc, so a freed arena slot is never reused and a missed root panics instead of reading another object. tests/stress_gc.rs runs the 95 programs in closure/, class/, inheritance/, string/, function/, super/, method/ and field/ with stress_gc, a collection before every allocation. With rustc 1.97 both end ok: 246 passed, and 95 passed.

The stronger check: a diff

Comments check what a test’s author thought to write down. A diff checks everything. As in chapter 163, I ran all 265 programs, benchmarks included, through drip-rs and through drip --vm, saved stdout, stderr and the exit code of each, and compared the files with cmp. stderr and exit codes matched for all 265. stdout matched for 255. The other ten are benchmarks that print times measured with clock(), such as fib.drip’s print clock() - start;, plus zoo_batch.drip, which counts how much work it finishes in ten seconds. In those ten, every line that doesn’t come from the clock matched too.

Your turn

#[derive(Debug, Default, PartialEq)]
struct Expect {
    output: Vec<String>,
    errors: Vec<String>,
    runtime: Option<(String, usize)>,
    exit: i32,
}

// The rest of `line` after the first
// `marker`, if it has one.
fn after<'a>(
    line: &'a str,
    marker: &str,
) -> Option<&'a str> {
    let i = line.find(marker)?;
    Some(&line[i + marker.len()..])
}

// `// [line N] Error...`, maybe with
// `c ` or `java ` before `line`. The
// line number and the error, if the
// expectation applies to `lang`.
fn at_line<'a>(
    line: &'a str,
    lang: &str,
) -> Option<(usize, &'a str)> {
    let mut r = after(line, "// [")?;
    let (w, x) = r.split_once(' ')?;
    if w == "c" || w == "java" {
        if w != lang {
            return None;
        }
        r = x;
    }
    let r = r.strip_prefix("line ")?;
    let (n, err) = r.split_once("] ")?;
    if !err.starts_with("Error") {
        return None;
    }
    Some((n.parse().ok()?, err))
}

// Read a test's expectations from its
// comments. `lang` is "c" or "java".
fn parse(
    src: &str,
    lang: &str,
) -> Expect {
    todo!()
}

#[derive(Debug, Default, PartialEq)]
struct Expect {
    output: Vec<String>,
    errors: Vec<String>,
    runtime: Option<(String, usize)>,
    exit: i32,
}

// The rest of `line` after the first
// `marker`, if it has one.
fn after<'a>(
    line: &'a str,
    marker: &str,
) -> Option<&'a str> {
    let i = line.find(marker)?;
    Some(&line[i + marker.len()..])
}

// `// [line N] Error...`, maybe with
// `c ` or `java ` before `line`. The
// line number and the error, if the
// expectation applies to `lang`.
fn at_line<'a>(
    line: &'a str,
    lang: &str,
) -> Option<(usize, &'a str)> {
    todo!()
}

// The VM's skip list: folders, or
// single files without ".drip".
const SKIP_VM: &[&str] = &[
    "scanning",
    "expressions",
    "benchmark",
];

// Skip `test` ("for/x.drip", say) if
// it is `p.drip` or is inside `p/`
// for some `p` in `skip`.
fn should_skip(
    test: &str,
    skip: &[&str],
) -> bool {
    todo!()
}

What you learned

• The shared corpus is 265 programs adapted from Crafting Interpreters’ MIT-licensed tests, read by all three implementations.
• Expectations live in comments: output, runtime errors, and compile errors, some only for c or java.
• tests/common/mod.rs ports check() without regexes, and two test crates share it as mod common;.
• harness = false gives one line per program and a substring filter. It must catch panics itself (catch_unwind), or one panic would stop the whole run.
• The VM suite runs 246 programs with debug_gc, and 95 more under the stress collector.
• A diff of all 265 against the TypeScript VM matched on stderr and exit codes everywhere, and on stdout everywhere except clock readings.