146. Tokens as enums

Forty names

Open companion/ts/src/token.ts and the token types are strings:

type TokenType =
  | '(' | ')' | '{' | '}' | ','
  // ...
  | 'identifier' | 'string' | 'number'
  | 'and' | 'class' | 'else' // ...
  | 'eof';

The VM’s scanner adds one more, type VmTokenType = TokenType | 'error', for forty in all. At run time each one is an ordinary JavaScript string, and the compiler tests them with current.type === 'identifier'. The union type is what makes a typo like 'identifer' a compile error.

Rust has no string literal types. It has something better suited to the job, and you met it in chapter 128.

TokenKind

Here is the start of drip-rs’s version, from scanner.rs:

#[derive(Clone, Copy, Debug,
         PartialEq, Eq)]
pub enum TokenKind {
    // Single-character tokens.
    LeftParen,
    RightParen,
    // ...
    Identifier,
    String,
    Number,
    And,
    Class,
    // ...
    Error,
    Eof,
}

Forty variants, the same forty as the TypeScript union, Error included. Two changes are forced. A variant must be an identifier, so '(' becomes LeftParen and '>=' becomes GreaterEqual. And Rust can’t extend an enum the way TokenType | 'error' extends a union, so Error lives in the one enum from the start.

The derive line asks the compiler to write four trait impls (chapter 131). Copy (with Clone) lets a kind be passed around like an integer. PartialEq and Eq give ==, which the compiler’s check needs:

fn check(&self, kind: TokenKind)
    -> bool {
    self.current.kind == kind
}

Debug prints a variant’s name, Let, for free. Chapter 148 will lean on that.

Token<‘src>

A token is a kind, the text it came from, and a line:

pub struct Token<'src> {
    pub kind: TokenKind,
    pub lexeme: &'src str,
    pub line: usize,
}

The TypeScript VmToken has the same three fields, but its lexeme is a new string made by source.slice(start, current). Here it is a &str slice: a pointer into the source and a length. Step through the tokens of one line:

Chapter 133 took this struct apart: 'src ties every token to the text it slices, and E0597 stops a token from outliving it. In drip-rs that never comes up, because the compiler holds the source for the whole compile, and its Compiler<'src, 'a> struct says so in its type. What this chapter adds is the size. A Token is 32 bytes on this machine: 1 byte of kind, a 16-byte slice and an 8-byte line, padded to a multiple of 8 (chapter 139). It is Copy, so the compiler’s previous and current are plain copies, not borrows of the scanner.

Error tokens

When the scanner meets @, it doesn’t stop. It returns a token of kind Error whose lexeme is the message, exactly as the TypeScript scanner does; chapter 133 showed why a &'static str message fits in the &'src str field. That one variant is why Error had to be in the enum from the start, where TypeScript could bolt 'error' on.

The same trick makes tokens the compiler invents. To compile this inside a method, or super in a subclass, drip-rs calls Token::synthetic("this", line): an Identifier whose lexeme is a literal, so the compiler can resolve it like any other variable name.

Keywords

The TypeScript scanner looks words up in a Map from text to type, KEYWORDS. drip-rs uses a match on the text instead:

fn keyword(text: &str)
    -> Option<TokenKind> {
    use TokenKind as T;
    Some(match text {
        "and" => T::And,
        "class" => T::Class,
        // ... 14 more
        _ => return None,
    })
}

Two small idioms here. use TokenKind as T gives the enum a short local name. And the _ arm says return None, which leaves the function, so every arm that finishes yields a TokenKind and the Some(...) wraps it. The caller turns “not a keyword” into an identifier:

let kind = keyword(text)
    .unwrap_or(TokenKind::Identifier);

There is no table to build at startup and nothing to allocate. And the match patterns are exact strings, so classy and Class both fall through to _.

#[derive(Clone, Copy, Debug, PartialEq)]
enum TokenKind {
    LeftParen,
    Identifier,
    Number,
    Fn,
    Let,
    Error,
    Eof,
}

#[derive(Clone, Copy, Debug)]
struct Token<'src> {
    kind: TokenKind,
    lexeme: &'src str,
    line: usize,
}

fn keyword(text: &str)
    -> Option<TokenKind> {
    use TokenKind as T;
    Some(match text {
        "fn" => T::Fn,
        "let" => T::Let,
        _ => return None,
    })
}

fn word<'s>(text: &'s str)
    -> Token<'s> {
    use TokenKind as T;
    let kind = keyword(text)
        .unwrap_or(T::Identifier);
    let line = 1;
    Token { kind, lexeme: text, line }
}

fn main() {
    let src = String::from("let fun");
    let a = word(&src[0..3]);
    let b = word(&src[4..7]);
    println!("{:?}", a);
    println!("{:?}", b.kind);
    let t = TokenKind::Let;
    println!("{}", a.kind == t);

    // An error token: its lexeme is a
    // &'static str message, not source.
    let e = Token {
        kind: TokenKind::Error,
        lexeme: "Unexpected character.",
        line: 3,
    };
    let (n, m) = (e.line, e.lexeme);
    println!("[line {n}] {m}");

    use std::mem::size_of;
    let k = size_of::<TokenKind>();
    let t = size_of::<Token>();
    println!("kind {k} B, token {t} B");
}

Your turn

#[derive(Clone, Copy, Debug, PartialEq)]
enum Kind {
    And, Class, Else, False, Fn, For,
    If, Let, Nil, Or, Print, Return,
    Super, This, True, While,
    Identifier,
}

fn keyword(text: &str) -> Option<Kind> {
    use Kind as K;
    Some(match text {
        "and" => K::And,
        "class" => K::Class,
        // ...the other fourteen
        _ => return None,
    })
}

fn word_kind(text: &str) -> Kind {
    todo!()
}

#[derive(Clone, Copy, Debug, PartialEq)]
enum Kind {
    LeftParen, RightParen, Bang,
    BangEqual, Less, LessEqual,
    Identifier, Str, Number,
    Fn, Error, Eof,
}

fn ts_name(k: Kind) -> &'static str {
    todo!()
}

What you learned

• drip-rs’s TokenKind is a token enum: forty fieldless variants, one per TypeScript token type, Error included. Punctuation gets names like LeftParen.
• #[derive(Clone, Copy, Debug, PartialEq, Eq)] makes kinds cheap to copy, comparable with == and printable by name. A kind is 1 byte.
• Token<'src> holds a kind, a line and a &'src str lexeme that points into the source. A token is 32 bytes and can’t outlive its source.
• Error and synthetic tokens use &'static str literals, which outlive any 'src.
• Keywords are a match on the text that returns Option<TokenKind>.