187. The Python plugin in Rust

Same columns, same reasons

This is most of ints_stay_ints.py, from the reference’s Python corpus (chapter 116 showed what the TypeScript engine makes of programs like it):

@dataclass
class Cell:
    count: int
    weight: float

cs = []
for i in range(300):
    cs.append(Cell(i, i / 2))
for i in range(len(cs)):
    cs[i].count += 1
    cs[i].weight *= 2

The Rust CLI, with --python-auto (long lines wrapped):

cs (line 9) {count, weight} → struct of
  arrays
  · loop at line 12 in <module>() touches
    2 of 2 fields (weight, count)
  · typed array('d') columns for weight;
    list columns for count (may hold
    non-floats)
  · about 300 elements
  · 2 of 5 element accesses proven in
    bounds

and in the container class it emits:

        self.count = []
        self.weight = _vsc_array('d')

weight is a float, so it gets a typed array('d'). count is an int, and an int stored in array('d') comes back as a float and prints as 1.0, so it gets a plain list. Every byte of the output, code and explain, is the TypeScript engine’s.

Four modules, four files

crates/frontend-python depends only on the core, the IR and indexmap, and each module is one file of engine/ts/packages/frontend-python:

tokens.rs   tokens.ts   INDENT/DEDENT
parser.rs   parser.ts   the subset, an AST
lower.rs    lower.ts    scopes, dataclasses
emit.rs     emit.ts     column classes

The crate root implements LanguagePlugin for PyPlugin, the trait from chapter 171. Its lower fails closed: if parse_python returns a PySyntaxError (a comprehension, a lambda, a try, a base class), it lowers an empty module, explain prints “No arrays of objects found.”, and emit returns the source unchanged. I checked with the corpus program unsupported_left_alone.py: the output is the input.

Indentation from bytes

The tokenizer reads src.as_bytes(). Everything it inspects is ASCII (other characters can only be inside strings and comments), so byte offsets cut the source where the reference’s UTF-16 offsets do. It keeps a stack of indentation widths and a bracket depth:

At the start of a line with depth == 0, it measures the width, with a tab moving to the next multiple of 8, and compares it with the top of the stack: wider pushes and emits INDENT, narrower pops and emits a DEDENT per level, and a width between two levels is “inconsistent indentation”. Inside brackets, a newline is just whitespace, so a call spread over lines stays one logical line.

One small closure fixed a real bug:

// `src.charAt(k)`: past the end there
// is no character.
let at = |k: usize| -> u8 {
    s.get(k).copied().unwrap_or(0)
};

The reference used to read src[i], which past the end of a JavaScript string is undefined. A regex test turns undefined into the string "undefined", so /[eE]/ matched, and in a file ending k = 1 with no line break, 1 became a float. A file ending k = 0x1 hung: /[0-9a-fA-F_]/ matches "undefined" forever. In Rust, s.get(k) is None past the end, and byte 0 matches none of the classes. The reference now uses charAt, and two corpus programs pin it down.

An arena that walks like objects

The reference’s parser builds plain objects, and much of lower.ts walks them generically: every key, in insertion order, skipping decorators. The port stores nodes in an arena, as chapter 170 did for VIR. A Node has start, end, line, its decorators and a Kind:

pub enum Kind {
    For {
        target: NodeId,
        iter: NodeId,
        body: Vec<NodeId>,
    },
    Call {
        func: NodeId,
        args: Vec<NodeId>,
        keywords: Vec<(String, NodeId)>,
    },
    // ... 27 more
}

There are no keys to iterate, so Ast::children spells out each kind’s children in the order the reference created the keys:

Kind::For { target, iter, body } => {
    out.extend([*target, *iter]);
    out.extend(body);
}
Kind::Call { func, args, keywords } => {
    out.push(*func);
    out.extend(args);
    out.extend(
        keywords.iter().map(|(_, v)| *v));
}

Why care about order, if a walk visits the same nodes? Because the walks emit things. When a statement isn’t modelled, the lowering collects the names it reads, in walk order, and emits a read for each. Those become VIR instructions, the analyses visit instructions in order, and the order of the reasons in explain follows. A walk in another order would make the same decisions and still fail parity.

Only literals, only one class

Python evaluates a dataclass default once, when the class is defined. The emitter copies the default’s text to every call site instead. That is the same only for a literal, so lower.rs checks:

fn is_literal_default(
    ast: &Ast,
    e: NodeId,
) -> bool {
    match ast.kind(e) {
        // An f-string reads variables.
        Kind::Constant(ConstValue::Str {
            fstring: true, ..
        }) => false,
        Kind::Constant(_) => true,
        Kind::UnaryOp { op, operand }
            if op == "-" || op == "+" =>
        {
            matches!(ast.kind(*operand),
                Kind::Constant(
                    ConstValue::Int(_)
                    | ConstValue::Float(_)))
        }
        _ => false,
    }
}

The first arm is a fix. The reference’s parser makes an f-string a Constant, so tag: str = f"k={K}" counted as a literal. After K = 2.0, the compiled program printed k=2.0 where the original printed k=1.0. Its node already carried an fstring flag; the check just never looked at it. In the Rust enum the flag is part of the pattern, so the fix is one arm.

The same kind of question found the last bug. After P = Twice, every P(...) was still compiled as a record of the dataclass. A dataclass is now modelled only when its class statement is the only module-level statement binding the name (module_defs, a count per name).

Five bugs, two test suites

The README lists the five reference bugs this port found, each fixed in both engines with a new corpus program:

1. A number literal at the end of the file (above).
2. f-string defaults copied to each call: a miscompile.
3. global g with no module-level g crashed the compiler.
4. len(ps[i]), with a field named length, crashed the emitter.
5. A rebound dataclass name: a miscompile.

The Python side is held to the reference two ways. Parity: 27 Python programs (the reference’s corpus, its Python benchmark, and six written for the port: CRLF, tabs, backslash continuations, semicolons, renamed columns) compiled in both bounds modes, 54 fixture pairs. And a python3 differential test, which runs each original and compiled program and compares stdout and the last line of stderr. When I ran cargo test, both passed, 54 of 54 and 27 of 27.

Your turn

type Id = usize;

enum Kind {
    Name(String),
    Num(f64),
    Assign { target: Id, value: Id },
    For {
        target: Id,
        iter: Id,
        body: Vec<Id>,
    },
    Call {
        func: Id,
        args: Vec<Id>,
        kws: Vec<(String, Id)>,
    },
}

#[derive(Default)]
struct Ast {
    nodes: Vec<Kind>,
}

impl Ast {
    fn add(&mut self, k: Kind) -> Id {
        self.nodes.push(k);
        self.nodes.len() - 1
    }
}

// Preorder, like the reference's walk:
// the names, in the order visited.
fn names(a: &Ast, id: Id) -> Vec<&str> {
    let mut out = Vec::new();
    let node = &a.nodes[id];
    if let Kind::Name(n) = node {
        out.push(n.as_str());
    }
    for c in children(a, id) {
        out.extend(names(a, c));
    }
    out
}

fn children(
    a: &Ast,
    id: Id,
) -> Vec<Id> {
    todo!()
}

What you learned

• frontend-python is four modules, one per TypeScript file, behind the same LanguagePlugin trait. Outside the subset it fails closed: the file is left unchanged.
• The tokenizer works on bytes, with an indentation stack and a bracket depth that turns newlines into whitespace.
• The arena AST has no keys, so Ast::children lists children in the reference’s key order: walk order becomes VIR order becomes explain order.
• Only literal defaults are modelled, and a dataclass only if its name is bound once. Columns are array('d') for proven floats.
• The port found five reference bugs. 54 fixture pairs and a python3 differential test hold the engines together.