29. Numbers, strings, names and keywords

The long tokens

Here is one line of Drip:

let total = 4.25;

The scanner from the last chapter handles = and ; already. But the other three tokens are longer than one or two characters, and each needs its own small loop: a number, a name chosen by the programmer, and the word let, which belongs to the language. This chapter writes those loops. They are the last pieces of Drip’s scanner, which lives in companion/ts/src/scanner.ts. The excerpts below are lightly simplified to fit a phone screen.

Numbers

A Drip number is one or more digits, optionally followed by a dot and more digits. So 7, 007 and 4.25 are numbers. The scanner knows a number has started when it sees a digit:

if (isDigit(c)) {
  while (isDigit(peek())) advance();
  if (peek() === '.' &&
      isDigit(peekNext())) {
    advance(); // the '.'
    while (isDigit(peek())) advance();
  }
  const text = source.slice(start, current);
  add('number', Number(text));
}

The interesting line is the dot test. When the scanner reaches a . after some digits, one character of lookahead isn’t enough to decide. 4.25 is one number, but in 4. the dot is not part of the number. In Drip, a dot can also mean “get a property”, as in point.x. So the scanner peeks two characters ahead, and only takes the dot if a digit follows it.

if (isDigit(c)) {
  while (isDigit(peek())) advance();
  if (peek() === '.' &&
      isDigit(peekNext())) {
    advance(); // the '.'
    while (isDigit(peek())) advance();
  }
  add('number', Number(lexeme()));
} else if (isAlpha(c)) {
  while (isAlphaNum(peek())) advance();
  add(KEYWORDS.get(lexeme())
      ?? 'identifier');
}

The last line hands the text to Number, JavaScript’s own converter, which turns '4.25' into a 64-bit float^[2]. Drip gets its number format for free from its host language. The token now carries two things: the lexeme '4.25', exactly as written, and the value 4.25 in its literal field.

Strings

A Drip string starts with " and runs to the next ". Unlike JavaScript, Drip has no escape sequences like \n, and a string may run across several lines:

case '"': {
  while (peek() !== '"' && !atEnd()) {
    if (peek() === '\n') line++;
    advance();
  }
  if (atEnd()) {
    error(line, 'Unterminated string.');
    break;
  }
  advance(); // the closing "
  // the value leaves out both quotes
  add('string',
    source.slice(start + 1, current - 1));
  break;
}

Two details matter. The loop still counts newlines, so every token after a long string still gets the right line number. And if the file ends before the closing quote, the scanner reports Unterminated string. instead of quietly inventing a quote. For "hi", the lexeme is the four characters "hi", quotes included. The literal value is the two-letter string hi.

Names

An identifier starts with a letter or an underscore, and continues with letters, digits or underscores. So total, _count2 and x1 are identifiers. 2x is not: the scanner sees the digit first, scans the number 2, and then starts a fresh identifier x.

const isAlpha = (c: string) =>
  (c >= 'a' && c <= 'z') ||
  (c >= 'A' && c <= 'Z') || c === '_';

// in the scanner's switch:
} else if (isAlpha(c)) {
  while (isAlpha(peek()) ||
         isDigit(peek())) advance();
  const text = source.slice(start, current);
  add(KEYWORDS.get(text) ?? 'identifier');
}

Keywords

Look at the last line of that code again. let, if and while are spelled exactly like identifiers. The scanner reads them with the same loop, and only then asks a table whether the word is special:

const KEYWORDS = new Map([
  ['and', 'and'], ['class', 'class'],
  ['else', 'else'], ['false', 'false'],
  ['fn', 'fn'], ['for', 'for'],
  ['if', 'if'], ['nil', 'nil'],
  ['or', 'or'], ['print', 'print'],
  ['return', 'return'],
  ['super', 'super'], ['this', 'this'],
  ['true', 'true'], ['let', 'let'],
  ['while', 'while'],
]);

Each keyword gets its own token type, named after itself. If the table has no entry, get returns undefined, and ?? falls back to 'identifier'.

Why not spot if as soon as the scanner sees an i and an f? Because of maximal munch. In let iffy = 1;, a scanner that stopped at if would produce the keyword if and then a stray fy. Reading the whole word first, then looking it up, gets iffy right.

Because keywords are reserved, let if = 3; is an error. The parser gets the token if where it needs an identifier, and Drip reports Error at 'if': Expect variable name. Notice also that true, false and nil are keywords to the scanner. The parser, a few chapters on, turns them into literal values.

print 42;
print 4.250;
print "Drip strings
can span lines";
let iffy = "a name, not if";
print iffy;
let _count2 = 7;
print _count2;
print true;
print nil;

// Try: let if = 3;
// Try: print 2x;

Your turn

const isDigit = (c) =>
  c >= '0' && c <= '9';

function scanNumber(s, i) {
  // your code here
}

console.log(scanNumber('4.25;', 0));
console.log(scanNumber('7.x', 0));

const KEYWORDS = new Map([
  ['fn', 'fn'], ['if', 'if'],
  ['let', 'let'], ['while', 'while'],
]);
const isAlpha = (c) =>
  (c >= 'a' && c <= 'z') ||
  (c >= 'A' && c <= 'Z') || c === '_';
const isDigit = (c) =>
  c >= '0' && c <= '9';

function scanWord(s, i) {
  // your code here
}

console.log(scanWord('iffy = 1', 0));

What you learned

• A literal is a value written in the source. Its token keeps the lexeme ('4.25') and the literal value (4.25).
• Numbers need two characters of lookahead: the dot is taken only if a digit follows it.
• A string’s value leaves out the quotes. The scanner counts newlines inside it and reports a string that never ends.
• An identifier is a letter or _, then letters, digits and _.
• A keyword is scanned like an identifier, then found in a reserved-word table. Reading the whole word first is maximal munch.

The scanner is finished. Next: rules for which token sequences make sense.