34. Syntax errors and recovery

Three typos, one run

Say you have written twenty lines of Drip, and three of them have a typo. One parser stops at the first and tells you about it. You fix it, run again, meet the second, fix that, run again. Another parser tells you all three at once. A third tells you about the three, and then about forty more that are not really there, because it got lost after the first.

You want the second parser. This chapter is about how to be it.

What a syntax error is

In the Drip parser, every place where the grammar requires a particular token goes through consume:

const consume = (t, message) => {
  if (check(t)) return advance();
  throw error(peek(), message);
};
// e.g. at the end of a print statement:
consume(';', "Expect ';' after value.");

error reports the problem right away, then hands back a ParseError for consume to throw. The report always names the token the parser was looking at. For print 1 +; and for print 1 + (no ; at all), Drip says:

[line 1] Error at ';': Expect expression.
[line 1] Error at end: Expect expression.

The format is [line N] Error at 'lexeme': message, or at end when the parser ran out of tokens (errors.ts). Note that the place named is where the parser noticed. A missing ; at the end of line 1 is usually reported at the first token of line 2.

Here is a program with three broken lines. Run it and read the errors: one per line, all from a single run.

print 1 +;
print (2;
let = 3;

Don’t stop at the first

Reporting one error is easy: throw, and let the whole parse fail. Reporting several is harder, because after an error the parser is in the middle of something it no longer understands. If it carried on from exactly where it stopped, it might treat the rest of a broken statement as the start of a new one, and report errors that are only echoes of the first. Those are called cascading errors, and they send people hunting for problems that do not exist.

Panic mode

Drip uses the oldest and simplest strategy. The ParseError thrown by consume unwinds all the way out of whatever was being parsed, up to declaration(), the function that parses one statement:

// simplified from parser.ts
function declaration(): Stmt | null {
  try {
    if (match('let')) return letDecl();
    // … fn, class, statement()
  } catch (e) {
    if (!(e instanceof ParseError)) throw e;
    synchronize();
    return null; // drop this statement
  }
}

Then synchronize() throws tokens away until it reaches a point where a new statement probably starts: just after a ;, or just before a keyword that begins a statement (let, fn, class, if, while, for, print, return). In the real code that keyword check is a switch; here it is a set called STARTS. Step through it on a four-line program with two mistakes:

function synchronize() {
  advance(); // skip the bad token
  while (!atEnd()) {
    if (previous().type === ';') return;
    if (STARTS.has(peek().type)) return;
    advance();
  }
}

The price is visible in the step-through: print x; was skipped without being checked. If it had contained a mistake, that mistake would only show up on the next run. Panic mode accepts that, because a missed error costs you one more run, while a fake error costs you confusion.

Not every error needs panic. When the parser reads a + b = 3, it parses a + b, sees the =, and finds that the target is not a name. It reports Invalid assignment target. but does not throw. The parser knows exactly where it is, so it simply carries on.

A syntax error stops everything

What happens to the statements that parsed fine? They are never run. Here is the top of run in companion/ts/src/drip.ts:

const tokens = scan(source, reporter);
const program = parse(tokens, reporter);
if (reporter.hadError)
  return { stdout, stderr, exitCode: 65 };
// … only now does anything run

The scanner and the parser share one reporter, so an unterminated string from the scanner and a missing ; from the parser both set hadError. If anything went wrong, Drip prints the errors and stops with exit code 65, the code its test suite uses for “this program is not valid Drip”. Half a program is never run.

Your turn

The core exercise writes synchronize itself, over an array of lexemes. The extra one is a broken Drip program for you to repair, using the errors Drip gives you.

const STARTS = new Set(['class', 'fn',
  'let', 'for', 'if', 'while', 'print',
  'return']);

function sync(ts, i) {
  // your code here
  return i;
}

let name = "Drip"
print "Hello, " + name;
let n = (2 + 3;
print n * 2;

What you learned

• A syntax error is a token the grammar did not allow. Drip reports the line, the token where it noticed, and what it expected.
• Error recovery lets a parser report several real errors in one run, without cascading errors that are only echoes.
• In panic mode, the parser abandons the statement, skips tokens to a synchronization point (after a ; or before a statement keyword), and starts again.
• It always skips at least one token, so it can never get stuck.
• Some errors, like an invalid assignment target, are reported without panicking.
• If any error was reported, Drip runs nothing and exits with code 65.

Next, at last, the tree does something: we evaluate it.