TokenKind:
| "comma"
| "colon"
| "lparen"
| "rparen"
| "lbracket"
| "rbracket"
| "lbrace"
| "rbrace"
| "star"
| "dot"
| "dotdot"
| "dash"
| "hashInt"
| "hashId"
| "dollardollar"
| "eq"
| "neq"
| "gt"
| "lt"
| "gte"
| "lte"
| "literal"
| "path"
| "int"
| "bareword"
| "transformLetter"
| "eof"
MQL lexer — tokenizes raw query text (after the desugar pass) into a stream of typed Tokens consumed by the parser.
Reserved character set (per req §1):
,:()[].-#$*/'=!=><>=<=Tricky cases:
Hyphen (
-) is set difference between expressions, but a literal character inside a bareword. The lexer absorbs-into a bareword only when both the previous and next chars are bareword chars; standalone-is the TokenKind.Dash operator.Hash (
#) is shape-dispatched:#5lexes as TokenKind.HashInt (chain-position ordinal);#abc123lexes as TokenKind.HashId (stuff id seed). A bare#not glued to digits or an identifier is a lex error.Single-quoted literals (
'…') preserve whitespace and have no escape syntax — the closing quote is the only terminator. An unterminated literal is a lex error.Paths start with
/and absorb identifier-shaped characters plus*,?, and-. The path stops at whitespace,., or any reserved character that isn't legal inside a path segment.Numbers vs. identifiers: the lexer scans a "word" greedily (digits, letters, underscores, with
-mid-word) and then classifies as TokenKind.Int when every character is a digit, TokenKind.Bareword otherwise. So5thlexes as one bareword,5as an int,oak-dooras one bareword.