24. Benchmarking honestly

A stopwatch is not enough

Here is the benchmark almost everyone writes first:

const t0 = performance.now();
sumAll(xs);
const t1 = performance.now();
console.log(t1 - t0, 'ms');

It looks reasonable. It is wrong in four separate ways. It times the function the very first time it runs, before the engine has optimized it. It throws the result away, so a clever compiler might skip the work. It trusts one measurement on a machine that is also doing a hundred other things. And it may run straight after some other benchmark that left the engine in a strange state.

This chapter takes the four problems one at a time, using the real harness behind every chart in this part of the book.

Warm-up: time the code that really runs

A JavaScript engine doesn’t run your function the same way every time. V8 first compiles all code to bytecode and interprets it. While it runs, V8 records how the program behaves, including object shapes and types. Functions that get hot are compiled by faster tiers (Sparkplug, then the optimizing compilers Maglev and TurboFan), which use that feedback to generate specialized machine code^[1]. That code is speculative: if an assumption breaks, V8 throws it away and falls back, which is called deoptimization^[1].

So the first calls of a function measure the interpreter and the compilers, not your loop. The harness runs the work for 300 ms first, and at least five times, before recording anything. Step through its core:

function measure(run) {
  let sink = 0;
  const end = now() + 300; // warm up
  while (now() < end) sink += run();
  const ts = [];
  for (let s = 0; s < 30; s++) {
    const t0 = now();
    sink += run();
    ts.push(now() - t0);
  }
  ts.sort((a, b) => a - b);
  return { median: q(ts, 0.5),
    p95: q(ts, 0.95), sink };
}

Dead code: make the result matter

Compilers delete work whose result nobody uses. It’s a standard optimization called dead-code elimination; LLVM, for example, ships several passes that do it^[2]. A loop whose answer is thrown away is a perfect candidate, and an optimizing JIT that removes it would make your benchmark report a near-zero time for a loop that never ran.

The harness’s defence is simple. Every kernel returns a number. Every returned number is added into a variable called sink, and at the end of each case the sink is printed. Now the work has an observable result, so it can’t be deleted:

let sink = 0;
function timeIt(run) {
  const t0 = performance.now();
  sink += run(); // keep the result
  return performance.now() - t0;
}
// ...later: print sink

Median and p95: summarize honestly

Even warm code doesn’t take the same time twice. A garbage-collection pause, another program, or the operating system can make one sample much slower. Look what one bad sample does to the mean:

const ts = [2.1, 2.0, 2.2, 2.0,
  30, 2.1, 2.3];
let sum = 0;
for (const t of ts) sum += t;
console.log('mean', sum / ts.length);

const s = [...ts].sort((a, b) => a - b);
console.log('sorted', s);
console.log('median', s[3]);

The harness takes 30 samples, sorts them, and reports two numbers. The median (the 50% point) says what a typical run costs. The p95 (95% of samples were at most this slow) says how bad the slow end gets. Each sample also lasts at least 10 ms, because one call of a fast loop is too short to time: browsers deliberately round performance.now() to 100 microseconds, or 5 in specially isolated pages^[3].

One fresh process per case

The last trap is the sneakiest. V8’s optimized code is specialized for the shapes and types it has seen^[1]. Suppose the get-objects suite ran every layout in one process. The shared consume(p) function would see object literals, then instances of one class, then of another: three different shapes. Its feedback becomes mixed, so each layout’s time would depend on which layouts ran before it.

So the harness never runs two cases in the same process. Its driver starts a brand-new Node process for every combination of suite, layout and size:

for (const layout of LAYOUTS) {
  const out = execFileSync(
    process.execPath,
    ['case.ts', suite, layout, n],
  );
  rows.push(JSON.parse(out));
}

A fresh process also starts with a fresh heap, so garbage left by one layout can’t trigger a collection during the next. The in-browser LiveBench in chapter 23 does the same with a fresh Web Worker per layout.

Noise is normal

This book’s results came from a shared cloud machine, and the results file says so: expect noise, and compare ratios, not absolute times. A ratio like “SoA was about 4× faster” survives a slower machine; “1.7 ns” does not. When a result surprises you, rerun it, and look at the p95 next to the median: a wide gap means the run was noisy.

Your turn

The core exercise hands you the broken stopwatch from the start of the chapter. Fix it.

function bench(fn, now) {
  const t0 = now();
  fn();
  const t = now() - t0;
  return { median: t, sink: 0 };
}

function p95(xs) {
  // your code here
}

console.log(p95([2, 1]));

What you learned

• JIT warm-up: V8 interprets first and optimizes hot code later, so warm up before measuring.
• Dead-code elimination can delete work whose result is unused; a sink keeps the result observable.
• Summarize many samples with the median (typical) and p95 (slow end), not one run or the mean.
• Benchmark isolation: one fresh process per case, so feedback and heap state don’t leak between layouts.
• On shared machines, trust ratios and look at the spread.

That’s the end of Part II. Next comes Part III: you’ll start building Drip, your own programming language, one piece at a time.