64. Inline caches

One line, any object

Here is about the smallest function that reads a property, and the bytecode Ignition makes for it:

// function getX(o) { return o.x; }
// node --print-bytecode
//   --print-bytecode-filter=getX
// (addresses and raw bytes removed)
@ 0 : GetNamedProperty a0, [0], [0]
@ 4 : Return

a0 is o. The first [0] is entry 0 of the constant pool, the name "x". The second [0] is a feedback slot. V8’s source describes this bytecode in one line: it “calls the LoadIC at FeedBackVector slot <slot>”^[1]. IC stands for inline cache.

The problem it solves: getX might receive any object at all. Last chapter you saw that where x lives depends on the object’s shape. Doing the full search through the shape’s descriptors on every call would be slow. So each access remembers what it found last time.

Remember what you saw

An inline cache keeps, for one access in the code, pairs of shape and how to get the property for that shape: say, “shape A: load slot 0”. When an object arrives, it checks the object’s shape against what it remembers. If it matches, it does the remembered load. If not, it does the slow lookup and remembers the new answer^[2]. Step through one access as five different shapes arrive:

Watching a real cache

Here is the program from the diagram. Five shapes, each with x first, and ten calls each:

function getX(o) { return o.x; }
const os = [
  { x: 1 }, { x: 1, a: 0 }, { x: 1, b: 0 },
  { x: 1, c: 0 }, { x: 1, d: 0 },
];
for (const o of os) {
  for (let i = 0; i < 10; i++) getX(o);
}

V8 has a flag, --log-ic, that logs IC state transitions to a file. (You may see --trace-ic mentioned online; this Node build refuses it, with node: bad option: --trace-ic.)

// node --log-ic --logfile=ic.log ic.js
// Node v22.23.4, V8 12.4.254.21
// kind, line:column, old, new, name
LoadIC 1:29 0 1 x
LoadIC 1:29 1 P x
LoadIC 1:29 P P x
LoadIC 1:29 P P x
LoadIC 1:29 P N x

Line 1, column 29 is the x in o.x. Its state went 0 → 1 on the first shape, 1 → P on the second, stayed P for the third and fourth, and went P → N on the fifth. The letters come straight from V8’s source^[2].

Why the optimizing tiers care

The feedback isn’t only for Ignition. Sparkplug keeps the feedback vector in its stack frame, because most operations need it^[6]. And in 2023 the V8 team described what Maglev does with a monomorphic o.x: it emits a check that o still has that shape, then a plain load at a fixed offset. After that it knows the shape, and doesn’t check it again^[5]. A megamorphic site gives it nothing to specialize on. The idea is old: Hölzle, Chambers and Ungar introduced polymorphic inline caches for Self in 1991, and noted that the receiver types a cache records are information an optimizing compiler can use^[9].

The word inline is historical. V8’s old baseline compiler patched the fast path into its machine code. Since Ignition (Chrome 59), the ICs are data-driven: the remembered shapes live in feedback vectors, separate from the bytecode. That separation is also what let V8 cache bytecode after a script has run, as in chapter 62^[4].

Keeping sites monomorphic

Megamorphic sites appear in ordinary code. A helper called with many kinds of objects is one. The V8 team’s 2021 post on super shows another: a super.m() call inside a mixin, where each use of the mixin creates a new class, makes that one access site megamorphic^[8]. Try it on your device:

function sumA(xs) {
  let s = 0;
  for (const o of xs) s += o.x;
  return s;
}
function sumB(xs) {
  let s = 0;
  for (const o of xs) s += o.x;
  return s;
}
const one = [], many = [];
for (let i = 0; i < 1000; i++) {
  one.push({ x: i, k: 0 });
  const o = { x: i };
  o['k' + (i % 20)] = 0; // 20 shapes
  many.push(o);
}
function time(sum, xs) {
  const t = performance.now();
  for (let r = 0; r < 3000; r++) {
    sum(xs);
  }
  const ms = performance.now() - t;
  return ms.toFixed(1) + ' ms';
}
for (let k = 0; k < 2; k++) {
  console.log('1 shape: ',
    time(sumA, one));
  console.log('20 shapes:',
    time(sumB, many));
}
// The numbers are your device's.

Your turn

// Simulated objects: { shape, vals }.
const shapes = new Map();
function make(keys, vals) {
  const id = keys.join();
  if (!shapes.has(id)) {
    shapes.set(id, { keys });
  }
  const shape = shapes.get(id);
  return { shape, vals };
}
let lookups = 0;
function slowLookup(o, key) {
  lookups++;
  return o.shape.keys.indexOf(key);
}

function makeIC(key) {
  const ic = (o) => {
    ic.misses++;
    return o.vals[slowLookup(o, key)];
  };
  ic.hits = 0;
  ic.misses = 0;
  return ic;
}

function icStates(seq) {
  const seen = new Set();
  return seq.map((s) => {
    seen.add(s);
    return seen.size === 1 ? '1' : 'P';
  });
}

What you learned

• An inline cache remembers, at one property access, the shapes it has seen and how to read the property for each.
• A hit is a shape check plus a load; a miss does the full lookup and updates the cache.
• Monomorphic means one shape, polymorphic a few (up to four in the V8 inside Node 22), megamorphic too many, using a shared stub cache.
• V8’s ICs live in feedback vectors, and Maglev and TurboFan turn monomorphic feedback into a check plus a direct load.
• Keep hot access sites monomorphic by making their objects the same way.

Next: arrays get their own version of shapes, elements kinds.