169. The Rust workspace

The same compiler, twice

In Part VII you built Viscosity in TypeScript, in engine/ts/packages. It reads a program, lowers it to VIR, decides which arrays of objects to turn into columns, and rewrites the source. Part XI walks through engine/rust, the same compiler written in Rust, the way Part X walked through drip-rs.

The port has one rule above the others. Run both engines on the same program with the same options, and the rewritten code and the explain text must be equal, byte for byte. Not equivalent. Equal.

That last part really happened. The README lists ten bugs that porting found in the TypeScript engine, three of them soundness bugs: the compiled program printed something the original didn’t. Every one is now fixed in both engines.

Six crates

Chapter 135 drew the workspace’s crates and what depends on what. Here they are next to the TypeScript packages they port:

The README pairs every Rust module with one TypeScript file: vir.ts becomes the vir crate, plan.ts becomes core::plan, lower.ts becomes frontend_js::lower. When a later chapter shows Rust, you already know which file to hold it up against.

The root manifest

The root Cargo.toml holds no code. Trimmed:

[workspace]
members = ["crates/vir", "crates/core",
    "crates/frontend-js",
    "crates/frontend-python",
    "crates/cli", "crates/wasm"]

[workspace.package]
version = "0.1.0"
edition = "2021"
rust-version = "1.97"

[workspace.dependencies]
viscosity-vir = { path = "crates/vir",
    … }
oxc_parser = "0.156"
wasm-bindgen = "=0.2.106"
indexmap = "2"
rustc-hash = "2"

[profile.release]
lto = "thin"
codegen-units = 1

A member opts in key by key. crates/core/Cargo.toml says version.workspace = true and indexmap.workspace = true, and gets the root’s value^[1]. The trimmed … in the viscosity-vir entry is version = "0.1.0": cargo uses the path inside the workspace and the version once the crate is packaged, which chapter 198 needs^[2].

The version requirements say different things. "2" accepts any 2.x. "0.156" is narrower: before 1.0, cargo treats each minor version as incompatible, so it means at least 0.156.0 and below 0.157.0^[2]. That is chapter 119’s caret rule, and it matters here because oxc’s AST changes between releases. "=0.2.106" accepts exactly one version. The wasm-bindgen command-line tool refuses a .wasm file built with a library whose “schema version” differs from its own^[3], so the README pins the library to the CLI it builds with.

Profiles live only in the root manifest; cargo ignores them in members^[1]. The release profile asks for “thin” link-time optimization, which optimizes across crates but runs much faster than the full kind^[5], and for one codegen unit, since splitting a crate into more units may produce slower code^[4]. Chapter 142 met the same two keys in drip-rs.

.cargo/config.toml

One more file changes how the workspace builds:

# Debug info off: the build is much
# smaller and faster to link.
[profile.dev]
debug = 0

[profile.test]
debug = 0

This is not a manifest. It is cargo’s configuration, which cargo looks for in the directory you run it from and in each parent directory^[6], and its [profile] tables override the manifest’s^[6]. debug = 0 means no debug information, where the dev profile’s default is full debug information^[5]. The README says this keeps target/ several times smaller. The price: in a backtrace, your own functions appear without file and line (the panic message itself still names its line; I checked with rustc 1.97).

What the crates pull in

core adds two dependencies. Both are about the same TypeScript habit.

A JavaScript Map visits keys in insertion order. The analyses iterate over maps and sets, and that order decides the order of the lines in viscosity explain. Rust’s HashMap visits keys “in arbitrary order”^[7], and it is seeded randomly, so the order can change from one run to the next:

use std::collections::HashMap;

fn main() {
    let names = ["ps", "qs", "rs",
        "ts", "us", "vs"];
    let mut m = HashMap::new();
    let mut order = Vec::new();
    for (i, n) in
        names.iter().enumerate()
    {
        m.insert(*n, i);
        order.push((*n, i));
    }
    // insertion order, like a JS Map
    for (n, i) in &order {
        print!("{n}={i} ");
    }
    println!();
    // HashMap: arbitrary order
    for (n, i) in &m {
        print!("{n}={i} ");
    }
    println!();
}

So wherever order is observable, the port uses IndexMap and IndexSet, whose iteration order doesn’t depend on the keys’ hashes^[8]: they keep insertion order, as Map does. Where it isn’t observable, the port wants speed. rustc-hash is the non-cryptographic hash that rustc itself uses: std’s default hash resists denial-of-service attacks, which a compiler doesn’t need to worry about^[9]. The port imports it under std’s names:

use rustc_hash::{FxHashMap as HashMap,
    FxHashSet as HashSet};

The parity test

Here is how “byte for byte” is checked. scripts/gen_expected.mjs runs the TypeScript engine on every program: the Part VII corpus, the end-to-end benchmarks, the Python corpus, and programs written for the port in tests/extra. It writes each result to tests/expected, twice: once with the default options and once with --bounds assume.

corpus_basic_soa.js
corpus_basic_soa.explain.txt
corpus_basic_soa.assume.js
corpus_basic_soa.assume.explain.txt

crates/cli/tests/parity.rs compiles the same programs with the Rust engine and compares strings. On a mismatch it prints the first line where the two differ, which is the exercise below. When I ran cargo test -p viscosity-cli --test parity for this chapter (cargo 1.97), it printed parity: 178/178 (54 Python); at commit 288e497 it prints parity: 190/190 (54 Python), because programs keep being added. Other tests in the file check that every program has fixtures and that the viscosity binary prints the same thing as the library.

Your turn

fn first_diff(want: &str, got: &str)
    -> Option<usize> {
    todo!()
}

What you learned

• viscosity-rs is engine/rust: six crates that port the TypeScript engine module by module, held to byte-for-byte parity.
• The root Cargo.toml shares versions and dependencies with workspace = true, key by key. "0.156" means 0.156.x only, and "=0.2.106" means exactly that version.
• .cargo/config.toml is read from the directory you run cargo in and its parents, and can override profiles.
• Order is observable in explain, so the port uses insertion-ordered IndexMap where TypeScript used Map, and fast FxHashMap elsewhere.
• The fixtures are the TypeScript engine’s own output; parity.rs compares, and reports the first differing line.