189. Compiling the engine to WebAssembly

Once by hand, then every time

In chapter 143 you ran two commands, cargo build for the wasm32-unknown-unknown target and wasm-bindgen --target nodejs, and the Rust engine compiled JavaScript from inside Node. That was enough to check parity. It isn’t enough to ship. A browser needs different glue from Node. The file should be as small as it can be, because readers download it. And the wasm-bindgen command-line tool must be the exact version the crate was built with (chapter 169), or it refuses the file.

The Rust ecosystem has a standard tool for all of this: wasm-pack. wasm-pack build “creates the files necessary for JavaScript interoperability and for publishing a package to npm”: it compiles the crate, runs wasm-bindgen, and writes a pkg folder holding the .wasm, the JavaScript wrapper, your README and a package.json^[1]. For release builds it also runs wasm-opt, an optimizer from the Binaryen project^[1][4].

Why Viscosity builds by hand

Viscosity doesn’t use wasm-pack. Its package is built by engine/rust/packages/wasm/build.mjs, 140 lines of Node. The odd wall is the one-target rule. @viscosity/wasm wants both a nodejs build and a web build in a single package, and lets each environment pick its own (that is chapter 190). The script also writes the licence texts, and with --book copies the web build into this book. Its steps, trimmed:

// 1. cargo
cargo build --release
  --target wasm32-unknown-unknown
  -p viscosity-wasm
// 2. wasm-bindgen, at the locked version
for (const t of ['nodejs', 'web'])
  wasm-bindgen --target t
    --out-dir dist/... viscosity_wasm.wasm
// 3. wasm-opt, if it can be found
wasm-opt -O3 <features> f -o f
// 4. LICENSE, THIRD_PARTY_LICENSES.txt
// 5. --book: copy dist/web into the book

(The real code calls execFileSync for each command.) Step 3 is optional. With no wasm-opt the build carries on and says so, and the result works, just larger. The wasm-opt it finds first is an npm devDependency, so npm install is all a contributor needs.

What each step does to the file

I rebuilt commit c68daec from an empty target directory in /tmp, with cargo 1.97, wasm-bindgen 0.2.106 and wasm-opt 132, running the script’s commands one at a time. Cargo took 38 seconds on the shared VM, and each wasm-opt run about 25. Between steps I walked the section headers of the .wasm to see where the bytes were:

The engine has grown since. At commit 288e497, which adds oxc’s semantic checks (chapter 172), the same three steps give 1,844,560, 1,759,791 and 1,373,390 bytes: about 1.4 MB, 486 KiB gzipped.

A WebAssembly file is a list of numbered sections: types, functions, exports, code, data, and “custom” sections that engines may ignore. The run below walks a 52-byte module the same way, reading each section’s id and its size (a variable-length LEB128 number):

use std::str;

// Read one LEB128 number (7 bits a
// byte, high bit = more to come).
fn leb(
    b: &[u8],
    i: &mut usize,
) -> usize {
    let (mut r, mut s) = (0, 0);
    loop {
        let x = b[*i];
        *i += 1;
        r |= ((x & 0x7f) as usize) << s;
        s += 7;
        if x & 0x80 == 0 {
            return r;
        }
    }
}

fn kind(id: u8) -> &'static str {
    match id {
        0 => "custom",
        1 => "type",
        3 => "function",
        7 => "export",
        10 => "code",
        11 => "data",
        _ => "other",
    }
}

fn main() {
    // f() returns 42; a "name" section
    // calls it "answer".
    let m: &[u8] = &[
        0, 97, 115, 109, 1, 0, 0, 0,
        1, 5, 1, 0x60, 0, 1, 0x7f,
        3, 2, 1, 0,
        7, 5, 1, 1, 102, 0, 0,
        10, 6, 1, 4, 0, 0x41, 42, 11,
        0, 16, 4, 110, 97, 109, 101,
        1, 9, 1, 0, 6,
        97, 110, 115, 119, 101, 114,
    ];
    assert_eq!(&m[..4], b"\0asm");
    let mut i = 8;
    while i < m.len() {
        let id = m[i];
        i += 1;
        let n = leb(m, &mut i);
        let end = i + n;
        let mut w =
            String::from(kind(id));
        if id == 0 {
            let l = leb(m, &mut i);
            let b = &m[i..i + l];
            let s = str::from_utf8(b);
            let s = s.unwrap();
            w += &format!(" {s:?}");
        }
        println!("{w}: {n} bytes");
        i = end;
    }
}

The second step explains a mystery from chapter 143: how does a command-line tool know that compile takes a string and returns an Output? The #[wasm_bindgen] macro can’t write that down as data, because it runs before the compiler has resolved the types. So it emits code instead: one __wbindgen_describe_ function per export, which reports the types when run. The CLI contains “a tiny and incomplete Wasm interpreter” for exactly this. It runs every describe function, then deletes them all^[3].

Two kinds of glue

The two wasm-bindgen runs produce the same .wasm (64 bytes apart) and different JavaScript. The nodejs target writes CommonJS that can be required like any Node module^[2]. Its last lines, trimmed:

const wasmBytes = require('fs')
  .readFileSync(wasmPath);
const wasmModule =
  new WebAssembly.Module(wasmBytes);
const wasm = new WebAssembly.Instance(
  wasmModule, imports).exports;

That is synchronous: compile works the moment require returns. A browser can’t read files, so the web target writes an ES module with an init() that you await. It fetches the .wasm (by default, from next to the glue) and instantiates it, streaming if it can^[2]:

import init, { compile }
  from '@viscosity/wasm/web';
await init();
const { code, explain } = compile(src);

Checking the tools

Step 2 begins with the check you’ll write below. It reads the wasm-bindgen version from Cargo.lock, asks the CLI for its own, and stops with the exact cargo install command if they differ. One more check is worth having for a committed binary. The web .wasm from my rebuild was byte-identical to the one book/public/wasm held then, and a rebuild at 288e497 matches today’s file, so the file this book serves really is the engine at that commit.

Your turn

const V: &str = "version = \"";

// The version Cargo.lock pins for the
// crate called `name`, if any.
fn locked<'a>(
    lock: &'a str,
    name: &str,
) -> Option<&'a str> {
    todo!()
}

// `out`: what the CLI's --version
// printed. Ok if it matches the lock.
fn check(
    lock: &str,
    out: &str,
) -> Result<(), String> {
    todo!()
}

What you learned

• wasm-pack compiles, binds, optimizes and packages a crate for npm, for one target at a time. build.mjs does the same steps by hand, for two targets.
• cargo’s file carries describe functions and a custom section for the CLI. wasm-bindgen runs them, writes the glue and removes them.
• wasm-opt -O3 shrank the code by about 8% and dropped about 200 KB of names. Gzipped, the whole saving was under 5%.
• The nodejs glue loads synchronously with readFileSync. The web glue needs await init().
• The CLI must match Cargo.lock exactly, and the committed .wasm rebuilds byte for byte.