25. A map of the territory

A string goes in, a 7 comes out

Here is a complete program in Drip, the small language you are about to build:

print 1 + 2 * 3;

Run it and you see 7. That feels like nothing. But think about what your program actually receives. Not a sum. Not even the word print. It receives a string, and you know from chapter 10 what a string is: a row of characters, 'p', 'r', 'i' and so on, numbered from 0.

Somewhere between that row of characters and the 7, a program has to notice that print is one word, that 2 * 3 happens before the +, and that the answer should be shown. Nobody does all that in one go. The work is split into stages.

The pipeline

Step through what happens to that one line. Watch the strip along the top: it shows which stage you’re in.

Each stage is just a function. It takes the previous stage’s output and returns something more organised. Here is the heart of Drip’s run, simplified from companion/ts/src/drip.ts:

const tokens = scan(source, reporter);
const program = parse(tokens, reporter);
const locals = resolve(program, reporter);
interp.interpret(program, locals);

Four calls, four modules, one job each, just as chapter 12 promised. The reporter collects error messages along the way. The resolve step is an extra check over the tree that you’ll meet in chapter 42.

The runner below sends your code through those same four calls. Change the numbers, add brackets, or add a second print.

print 1 + 2 * 3;
print (1 + 2) * 3;

Interpreters and compilers

The diagram showed two ways to finish the journey. They have names.

An interpreter runs the program itself. Part III’s interpreter walks the syntax tree, and every node it visits does something: an addition adds, a print prints.

A compiler doesn’t run anything. It reads a program and writes out the same program in another form. That form might be machine code for the processor, bytecode for a virtual machine, or source code in another language. Then something else runs the result.

The two ideas mix freely. Drip’s VM in Part IV is both: a compiler turns the source into bytecode, then an interpreter loop runs the bytecode. Try the same line on the other back end:

print 1 + 2 * 3;
print (1 + 2) * 3;

Front end and back end

Look at the pipeline again and you can cut it in two. The front end reads the program and works out what it says. The back end makes it happen. The front end cares about the language you write in: the scanner must know that fn and let are Drip words. The back end cares about the target: a tree-walker, a bytecode VM, or a processor.

The point of the cut is that the middle, the tree or IR, is shared. One front end can feed several back ends, which is exactly what Drip’s two implementations show. Production compilers go further, with a whole “middle end” of analysis and optimization passes between the two halves^[2].

That bottom row is where this whole book is heading. Viscosity is a source-to-source compiler: JavaScript in, JavaScript out. Your arrays of objects go in, and where it is safe, the same program comes out with the data stored as a struct of arrays in typed arrays, the layouts you measured in Part II. That idea comes from research on doing this transformation automatically^[4]. To build it you need every stage on this map, so Part III starts at the very front.

Your turn

The core exercise builds a complete, tiny interpreter: a scanner and an evaluator for sums like 1 + 2 - 4. The extra one turns the same evaluator into a compiler that outputs instructions instead of answers.

function scan(src) {
  return src.split(' ');
}

function evaluate(ts) {
  // your code here
}

function run(src) {
  return evaluate(scan(src));
}

console.log(run('1 + 2 - 4'));

function exec(code) {
  const st = [];
  for (const ins of code) {
    const [op, arg] = ins.split(' ');
    if (op === 'push') {
      st.push(Number(arg));
    } else {
      const b = st.pop();
      const a = st.pop();
      if (op === 'add') st.push(a + b);
      else st.push(a - b);
    }
  }
  return st.pop();
}

function compile(ts) {
  // your code here
}

const code = compile(['1', '+', '2']);
console.log(code, exec(code));

What you learned

• A compiler pipeline passes a program through stages: source → tokens → tree → output (or another form). Each stage is a function with one job.
• An interpreter runs a program directly. A compiler translates it into another form without running it. A bytecode VM does both.
• The front end understands the source language. The back end produces the result for a target. A tree or IR in the middle lets them be mixed and matched.
• Viscosity is a source-to-source compiler with an IR (VIR) in the middle. Drip is where you learn to build each stage.

Next: meet Drip itself, the language your pipeline will run.