66. Speculation and deoptimization

What does + mean?

Here is the smallest function that keeps an engine busy:

function add(a, b) {
  return a + b;
}

+ adds numbers, joins strings, and calls valueOf on objects. When Ignition runs add, its bytecode has one generic Add, and every call has to work out which of those cases it is in. Your Drip VM’s ADD does a smaller version of this: two strings, or two numbers, checked on every single add.

But in a real program, add is almost always called the same way. If the engine could see that, it could compile a version that only adds integers. It would just need a way out for the day someone passes a string.

Type feedback

While unoptimized code runs, V8 keeps track of how the program behaves, including the shapes and types each operation sees^[1]. It lives in a feedback vector attached to the function, with a slot for operations such as +. You can read that slot with %DebugPrint(add) after three phases: 20 calls with integers, one with 1.5, one with strings. Real output, Node v22.23.4 (V8 12.4):

$ node --allow-natives-syntax fb.js \
    | grep 'slot #0'
 - slot #0 BinaryOp BinaryOp:SignedSmall {
 - slot #0 BinaryOp BinaryOp:Number {
 - slot #0 BinaryOp BinaryOp:Any {

Small integers, then any number, then anything. Like elements kinds, the feedback only ever widens.

Speculation

Once add is hot, the optimizing compiler builds code from the feedback. It doesn’t prove the feedback will stay true. It speculates, and puts a check in front of each guess. Maglev’s post describes the pattern for o.x: if feedback says o always had one shape, it emits a node that checks the shape at run time, then a cheap load at a fixed offset^[1]. For arithmetic, TurboFan speculates that small integers stay small, using a checked add that deoptimizes if the result overflows^[3].

Some guesses need no check in the code at all. A global that has never changed can be built into Maglev’s machine code as a constant. Maglev registers a dependency instead, and if the global is ever changed, the runtime invalidates and deoptimizes that code^[1].

function add(a, b) {
  return a + b;
}
for (let i = 0; i < 1e5; i++)
  add(i, 1);
add('a', 'b');

Deoptimization

Here is the real thing. Warm up add with 100 000 integer calls, then call add('a', 'b'). Real output, Node v22.23.4, add lines only, wrapped and shortened with …:

$ node --trace-opt --trace-deopt d2.js
[marking … <JSFunction add …> for
  optimization to TURBOFAN, …,
  reason: hot and stable]
[completed optimizing … add …
  (target TURBOFAN)]
[bailout (kind: deopt-eager,
  reason: not a Smi): begin.
  deoptimizing … add …,
  bytecode offset 2, …]

--trace-deopt-verbose adds where it happened and what the deoptimizer built:

;;; deoptimize at <…/d2.js:2:12>
…
translating interpreted frame add
  => bytecode_offset=2, …

Line 2, column 12 is the +. To get back into Ignition, V8 needs to know, at that exact point, where every interpreter register’s value lives in the optimized code: in a machine register, on the stack, or not computed at all. So optimized code carries that map with it. Maglev attaches the interpreter’s frame state to every node that can deoptimize, and the deoptimizer uses it to fill in a fresh interpreter frame; TurboFan shares the same mechanism^[1].

After the bailout

A deopt also cleans up. The optimized code is no longer valid, so it is unlinked from the function, and the next call goes back to unoptimized code^[2]. The failed call ran in Ignition, which updated the feedback to Any. If add gets hot again, the new code is compiled from that wider feedback, with no Smi guess to fail. In a longer run of the same idea, --trace-opt shows add marked and compiled for TurboFan a second time after the bailout.

Getting this wrong is expensive. Crankshaft, V8’s optimizing compiler before TurboFan, too often reoptimized a function with the same assumptions that had just failed, giving endless optimization–deoptimization loops^[3].

Your turn

function makeAdd() {
  const st = {
    mode: 'interp',
    feedback: 'none',
    hot: 0,
    deopts: 0,
  };
  function add(a, b) {
    return a + b;
  }
  add.st = st;
  return add;
}

function widen(fb, a, b) {
  return fb;
}

What you learned

• Type feedback is what unoptimized code observes: shapes, operand types, call targets. In V8 12.4, the slot for + widened from SignedSmall to Number to Any.
• Speculation compiles for the feedback seen so far. Each assumption has a guard or a dependency.
• Deoptimization happens when a guard fails. The deoptimizer uses the saved frame state to rebuild an interpreter frame, and the call finishes in Ignition with the same result.
• After a deopt the code is unlinked and the feedback is wider, so the next optimization doesn’t repeat the guess. Repeating it is a deoptimization loop.