Syntax.Token

The token vocabulary of the prism-syntax-tokens-v1 artifact.

A fixed token’s wire kind is its source spelling, so TFixed carries the spelling rather than enumerating every keyword and operator; value-carrying and virtual layout tokens each get a dedicated constructor matching the grammar’s terminal aliases.

Types

`TokenKind`

type TokenKind
  = TFixed(String)
  | TIdent
  | TUid
  | TQual
  | TInt
  | TFloat
  | TChar
  | TStr
  | TIStart
  | TIMid
  | TIEnd
  | TVOpen
  | TVClose
  | TVSemi
  deriving (Eq)

A token kind: a fixed token by exact spelling, or one of the special value-carrying and virtual kinds.

`Token`

type Token = Token { kind: TokenKind, span: Span, value: Option(String) }

One token: its kind, its span in the embedded source, and, for value-carrying kinds, the decoded payload (escapes resolved, digit separators stripped). The original spelling is always recoverable from the span.

`Trivia`

type Trivia = TComment(Span) | TBlockComment(Span) | TBlank(Span)

One trivia event between semantic tokens: a line comment, a delimited comment, or a blank-line run.

Functions and Values

`kind_name`

kind_name : (Syntax.Token.TokenKind) -> String

The canonical wire name of a token kind, matching the compiler’s artifact vocabulary (fixed tokens spell themselves; specials use the grammar’s terminal aliases).

(kind_name(TIdent), kind_name(TVOpen), kind_name(TFixed("->")))

(ident, v{, ->)

`kind_of_name`

kind_of_name : (String) -> Syntax.Token.TokenKind

The token kind a wire name denotes. Any name that is not one of the special aliases is a fixed token spelling; the fixed vocabulary itself is pinned by the compiler’s own tests, not re-enumerated here.

(kind_of_name("uid") == TUid, kind_of_name("match") == TFixed("match"))

(true, true)

`trivia_name`

trivia_name : (Syntax.Token.Trivia) -> String

The wire name of a trivia event.

`trivia_span`

trivia_span : (Syntax.Token.Trivia) -> Syntax.Source.Span

The span a trivia event covers.

Keyboard shortcuts

Prism