172. Parsing with oxc

Two right answers

One of the programs written for the port, tests/extra/unicode_offsets.js, starts like this (trimmed; in sep each … is a U+2028 or U+2029, invisible line-break characters):

// Ünïcödé before the collection: …
/* U+2028 and U+2029 are line breaks … */
const label = 'héllo wörld 😀 𝒳';
const sep = 'a…b…c'; // …
const ps = [];

I parsed it with both parsers. acorn, the TypeScript engine’s parser, says the statement const ps = []; runs from offset 266 to 280. oxc, the Rust engine’s, says 286 to 300. Twenty apart, and both correct. And the parity test checks that both engines rewrite this file identically. This chapter is about how both can be true.

oxc

oxc, the Oxidation Compiler, is “a collection of high-performance tools for JavaScript and TypeScript written in Rust”: a parser, a linter, a formatter, a transformer, a minifier^[1]. The port uses its parser and AST crates, version 0.156. The parser has “three inputs (a memory arena, a source string, and a SourceType) and one return struct”^[2]. Here is the call in crates/frontend-js/src/lib.rs, trimmed:

let allocator = Allocator::default();
let options = ParseOptions {
    preserve_parens: false,
    parse_regular_expression: true,
    ..ParseOptions::default()
};
let ret = Parser::new(&allocator, source,
        SourceType::mjs())
    .with_options(options)
    .parse();
if let Some(e) =
    ret.diagnostics.errors().next() {
    return Err(syntax_error(source, e));
}
let program = ret.program;
let semantic = SemanticBuilder::new()
    .with_check_syntax_error(true)
    .build(&program);
if let Some(e) =
    semantic.diagnostics.errors().next() {
    return Err(syntax_error(source, e));
}

Line it up with chapter 92’s acorn adapter. SourceType::mjs() is acorn’s sourceType: 'module'. preserve_parens: false matches acorn’s default of not keeping parentheses as nodes, and parse_regular_expression checks regex literals, as acorn does. Errors come back as data in ret, not as an exception, so the plugin turns the first one into a CompileError; syntax_error writes it as message (line:column), the shape of acorn’s messages. The second check is the pitfall below.

A tree in an arena

ESTree (chapter 92) is plain objects with a type string. oxc’s tree is Rust types: a Statement enum, an Expression enum with dozens of variants, and a struct per node kind. One variant, from oxc_ast:

pub enum Expression<'a> {
    // ...
    BinaryExpression(
        Box<'a, BinaryExpression<'a>>),
    // ...
}

That Box is not std’s. It is oxc’s arena box: a pointer to a node in the Allocator. On this machine an Expression is 16 bytes, a tag and that pointer. The kinds are split finer than ESTree’s, too: ps[i].x is a StaticMemberExpression whose object is a ComputedMemberExpression, where ESTree has one MemberExpression with a computed flag. Chapter 173 maps them back.

'a is chapter 133’s lifetime, and here it names the arena. Every node borrows the allocator, so no node can outlive it. Step through what that means for the plugin:

oxc’s documentation spells out the deal. Objects in the arena “are never Dropped”; the memory is “released in bulk when the allocator is dropped”. To keep that from leaking, the arena refuses, at compile time, any type that implements Drop, such as a std::vec::Vec. That is why oxc ships arena versions of Box, Vec, String and HashMap^[3].

For the port this settles a design question. The TypeScript engine keeps ESTree nodes alive inside the VIR, in SourceRef.node. The Rust tree dies when lower returns, so the VIR keeps a NodeRef index into a side table of facts instead (chapter 173). Spans survive: an oxc Span is two u32s with no lifetime.

Bytes and code units

Now the two offsets. A JavaScript string is a sequence of UTF-16 code units, and acorn’s “character offsets” count those. A Rust String is UTF-8 bytes, and oxc’s spans count bytes. Before const ps, the file has accented letters (one code unit, two bytes each), 😀 and 𝒳 (two code units, four bytes each), and U+2028 and U+2029 (one code unit, three bytes each). Measured on the same file:

             acorn       oxc
             (UTF-16)    (UTF-8)
const ps     266         286
ps[i].x      417         442
ps[3].name   715         749

Each engine splices its own string with its own offsets. The TypeScript emitter does out.slice(0, e.start) + e.text + out.slice(e.end). The Rust one, from emit.rs:

out.replace_range(
    e.start as usize..e.end as usize,
    &e.text);

slice counts code units and replace_range counts bytes, so both cut before the same p of ps[i].x. The fixture shows the result: the same s += ps.x[i] + t.length; } // ✓ — done line from both engines.

The danger is mixing the two. Hand acorn’s 266..280 to the Rust string and you get "9, and a line ", from a comment two lines up. Land inside a character and Rust panics, because a string slice must start and end on a character boundary^[5]. When the port does need a JavaScript-style offset, lines::utf16_offset converts.

fn main() {
    let s = "é😀x = 1";
    println!("bytes:  {}", s.len());
    let n = s.chars().count();
    println!("chars:  {n}");
    let u = s.encode_utf16().count();
    println!("UTF-16: {u}");
    let x = s.find('x').unwrap();
    let ux =
        s[..x].encode_utf16().count();
    println!("x: byte {x}, unit {ux}");
    println!("{:?}", &s[x..]);
    // byte 3 is inside the 😀:
    let ok = s.is_char_boundary(3);
    println!("{ok}");
    // so &s[3..] would panic
}

Line numbers need care too, because oxc reports none. The plugin counts them itself in lines.rs, at the same breaks acorn counts: \n, \r\n, a lone \r, U+2028 and U+2029. tests/extra/annotation_line_breaks.js puts a U+2028 between two declarations to check it.

Your turn

// The byte offset in src of the
// character at UTF-16 offset off (what
// acorn and JavaScript's slice count).
// off is never inside a character; off
// at the end gives src.len().
fn byte_offset(src: &str, off: usize)
    -> usize {
    todo!()
}

What you learned

• oxc is a Rust toolchain for JavaScript. Its parser takes an arena, the source and a SourceType, and returns the tree with its errors as data.
• The tree lives in a bump arena: 'a ties every node to the Allocator, nodes are never dropped one by one, and the whole arena is freed at once. So the port keeps spans and facts, not the tree.
• oxc spans count UTF-8 bytes, acorn’s offsets UTF-16 code units. Each engine slices its own string with its own offsets, so both cut at the same characters.
• Line numbers are counted by the plugin at every ECMAScript line break.
• oxc’s parser leaves some early errors to a semantic pass, so the port runs that pass’s syntax checks too.