126. Slices and strings

A token is a piece of the source

In chapter 28 your scanner made a token for every word of a Drip program, and each token carried its lexeme: the exact text it came from. The TypeScript scanner gets that text with

lexeme: source.slice(start, current)

which hands you a new string value. But the text already exists. It is sitting right there in source, between start and current. All a token really needs is a way to say “those bytes, over there”.

Rust has a type for exactly that, and it builds on the last chapter: it is a borrow of part of a value.

Slices of arrays

Write a range in square brackets after a borrow:

let a = [10, 20, 30, 40, 50, 60];
let mid = &a[1..4];   // 20, 30, 40
let head = &a[..2];   // 10, 20
let tail = &a[3..];   // 40, 50, 60

The range 1..4 starts at 1 and stops before 4. Leave out a side and it runs to the start or the end. Step through what each one points at:

The real payoff is in function parameters. A function that takes &[i32] doesn’t care how long the array is, or whether it is getting all of it:

fn sum(xs: &[i32]) -> i32 {
    let mut total = 0;
    let mut i = 0;
    while i < xs.len() {
        total += xs[i];
        i += 1;
    }
    total
}

sum(&a), sum(&a[1..4]) and sum(&a[3..]) all work. A borrow of a whole array turns into a slice automatically.

Ask for a range that isn’t there, like &a[2..9], and the program stops at run time with a message (Rust calls this a panic):

range end index 9 out of range for
slice of length 6

fn sum(xs: &[i32]) -> i32 {
    let mut total = 0;
    let mut i = 0;
    while i < xs.len() {
        total += xs[i];
        i += 1;
    }
    total
}

fn main() {
    let a = [10, 20, 30, 40, 50, 60];
    let mid = &a[1..4];
    println!("len {}", mid.len());
    println!("mid {}", sum(mid));
    println!("all {}", sum(&a));
    println!("tail {}", sum(&a[3..]));
}

String and &str

You met String in chapter 124. It owns its text on the heap, and it can grow. A string slice, written &str, is a view of some text that somebody else owns. That text might live in a String, or in the compiled program itself: a string literal like "print" is a &str, which is why literals can’t be changed^[1].

let mut own = String::from("let");
own.push_str(" x");     // String: grows

let view: &str = &own;  // whole String
let kw = &own[0..3];    // "let"
let lit = "print";      // &str literal

let copy = lit.to_string(); // new String

Converting from String to &str is free: it is just a borrow. Going the other way, String::from(x) or x.to_string(), copies the bytes into a fresh allocation.

So which should a function take? Prefer &str. A &str parameter accepts literals and slices, and when you pass &own for a String, Rust converts the &String to a &str for you^[1].

fn lexeme(
    src: &str,
    start: usize,
    end: usize,
) -> &str {
    &src[start..end]
}

fn lexeme(
    src: &str,
    start: usize,
    end: usize,
) -> &str {
    &src[start..end]
}

fn main() {
    let s = String::from("let x = 10;");
    let name = lexeme(&s, 4, 5);
    let num = lexeme(&s, 8, 10);
    println!("{name} and {num}");

    let lit = "print";
    println!("{}", lexeme(lit, 0, 2));

    let mut own = String::from(name);
    own.push_str("_2");
    println!("{own} {}", own.len());
}

Slices keep their owner borrowed

A slice is a borrow, so last chapter’s rule applies. This doesn’t compile:

let mut s = String::from("let x;");
let word = &s[0..3];
s.clear();
println!("{word}");

clear needs to change s, so it takes a mutable borrow, and word is still reading. The error is the same E0502 (real rustc 1.97 output, trimmed):

error[E0502]: cannot borrow `s` as
  mutable because it is also borrowed
  as immutable

In TypeScript, the token’s lexeme would be its own string and the question never comes up. In Rust a slice is cheaper, and the price is that the source has to stay put while tokens point into it. For a scanner that is exactly right: the source doesn’t change while you scan it.

end byte index 2 is not a char boundary;
it is inside 'é' (bytes 1..3 of string)

pub struct Token<'src> {
    pub kind: TokenKind,
    pub lexeme: &'src str,
    pub line: usize,
}

lexeme: &self.source[
    self.start..self.current],

Your turn

fn is_ident(c: u8) -> bool {
    c.is_ascii_alphanumeric()
        || c == b'_'
}

fn take_ident(src: &str) -> &str {
    // your code here
    ""
}

fn sum_range(
    xs: &[f64],
    from: usize,
    to: usize,
) -> f64 {
    0.0
}

What you learned

• A slice &[T] is a borrowed window on a run of elements: a pointer and a length. &a[1..4] excludes its end, and every access is bounds-checked.
• Functions that take slices work on whole arrays and on parts of them.
• String owns growable UTF-8 text. &str borrows text that lives elsewhere, including string literals. Take &str in parameters.
• A slice keeps its owner borrowed, so the owner can’t be changed while the slice is in use.
• String lengths and indices are in bytes.

Next: structs. You have seen one already, Token. Time to write your own, and give them methods.