Syntax.Lex

A Prism-language reimplementation of the compiler’s raw token layer: exact UTF-8 tokenization, literal payload decoding, and interpolation splitting.

The whole layer is ordinary Prism. The Rust lex_raw pipeline remains the authoritative oracle; this module produces the same raw token stream (kind, byte span, and decoded value) and the same interleaved trivia (line comments and blank-line runs) so the two can be diffed. It is compared and reported, never used as a silent fallback.

Scope of this layer: the fixed-token vocabulary, identifiers, uppercase names, qualified paths, numbers, char and string literals, line comments, and blank-line trivia; the decoded value each token carries (identifier text, separator-stripped integers, normalized floats, unescaped chars and strings); and the interpolation split of a "a {x} b" literal into istart/imid/ iend segment tokens with the hole expressions re-lexed at their absolute source offsets. A nested string literal inside a hole is scanned with a brace-depth stack, so its own quotes and braces never end the outer token early. Bracket tracking and the offside layout virtuals are later layers; no layout virtual is synthesized here.

Types

`LexRaw`

type LexRaw = LexRaw { tokens: List(Token), trivia: List(Trivia) }

The raw token layer: the semantic token stream and the trivia interleaved between them, each keyed by absolute byte span into the lexed text.

`LexError`

type LexError
  = LxInvalid(Int)
  | LxEmptyHole(Int)
  | LxUntermHole(Int)
  | LxUntermStr(Int)
  | LxNumberSep(Int)
  deriving (Eq, Show)

A lexing failure, mirroring the compiler’s five LexError variants one for one: an invalid token, an empty interpolation hole, an unterminated hole, an unterminated string, and a misplaced numeric separator. Each carries the byte offset where scanning stuck. The variant, its stable code, and its rendered message travel as data (see lex_offset/lex_code/lex_message), never encoded into a message string.

Functions and Values

`lex_offset`

lex_offset : (Syntax.Lex.LexError) -> Int

The byte offset where a lexing failure stuck.

`lex_code`

lex_code : (Syntax.Lex.LexError) -> String

The stable, append-only diagnostic code a lexing failure carries, matching the compiler’s LexError::code (E7000 through E7004).

`lex_message`

lex_message : (Syntax.Lex.LexError) -> String

The rendered message, matching the compiler’s LexError Display strings byte for byte so a decoded diagnostic compares equal.

`lex_raw`

lex_raw : (String) -> Result(Syntax.Lex.LexRaw, Syntax.Lex.LexError)

Tokenize text into the raw token stream and its interleaved trivia, reproducing the compiler’s lex_raw layer: kinds, byte spans, decoded values, interpolation splitting, comments, and blank-line runs. No layout virtuals.

`lex_incomplete`

lex_incomplete : (String) -> Bool

Whether text fails to lex only because a string literal or interpolation hole runs off the end: the lexer-observable half of the compiler’s incomplete classification, telling an interactive reader to keep reading rather than to reject. Any other lexing failure, or a clean lex, is complete.

Keyboard shortcuts

Prism