refactor(syntax): Group SyntaxKind and unify token/terminal kinds under LexemeKind.

[orizi]

Summary

Restructures the generated SyntaxKind enum so that the token and terminal kinds are no longer two parallel, duplicated variant lists.

A terminal node and the token that backs it share one lexical identity, so both now reuse a single LexemeKind enum. Trivia tokens (whitespace, newlines, comments, skipped) move to TriviaKind, and enum missing-variants to MissingKind:

enum SyntaxKind {
    Token(LexemeKind),
    TriviaToken(TriviaKind),
    Terminal(LexemeKind),
    Missing(MissingKind),
    // ...other node kinds stay flat (ExprBinary, ExprList, ...)
}

The lexer/parser layer now operates on LexemeKind directly (LexerTerminal, peek().kind, Terminal::KIND, MissingToken, operator-precedence and skip-until predicates); SyntaxKind is reserved for green-tree node kinds. All kind enums are generated from the spec through a single classifier, so the lists can't drift apart again.

Debug/Display preserve the historical flat names (TerminalIdentifier, TokenWhitespace, ExprMissing), so diagnostics and golden tests are unaffected by the grouping. colored_printer::set_color now matches exhaustively over LexemeKind/ panic! catch-all (the long-standing "Can this be made exhaustive?"TODO).

Type of change

▎ None of the listed categories is "refactor"; this is an internal restructuring with no intended functional change (see below for the one user-visible diagnostic wording change).

• Bug fix (fixes incorrect behavior)
• New feature
• Performance improvement
• Documentation change with concrete technical impact
• Style, wording, formatting, or typo-only change
• Refactor / internal cleanup (no functional change)

Why is this change needed?

The flat SyntaxKind duplicated every lexical kind twice — once as TokenX and once as TerminalX (81 paired names) — with nothing tying the two lists together; they could (and did) drift. A terminal and its backing token are the sameould share one source of truth.
Separately, colored_printer::set_color was a non-exhaustive match ovpanic!, which crashed on perfectly valid token kinds it didn'tenumerate. Grouping the kinds lets that match be exhaustive over LexemeKind/TriviaKind, so the compiler now forces a coloring decision for any new token instead of allowing a latent panic.

What was the behavior or documentation before?

• SyntaxKind was a flat enum with parallel Token* and Terminal* varintly from the spec.
• colored_printer::set_color panicked (Unexpected syntax kind: ...) on token kinds missing from its match.
• The "missing token" diagnostic rendered the internal kind name, e.tifier.

What is the behavior or documentation after?

• SyntaxKind groups token/terminal/missing kinds into LexemeKind (shiaKind, and MissingKind; other node kinds stay flat. Thetoken/terminal duplication is gone.
• set_color matches exhaustively; no catch-all panic.
• The only user-visible change: that diagnostic now reads Missing token Identifier. (the LexemeKind name), which is friendlier than the old TerminalIdentifier.

Everything else (Debug/Display strings, all other diagnostics, golden test output) is unchanged.

Related issue or discussion (if any)

None.

Additional context

• All three generated files (kind.rs, ast.rs, key_fields.rs) and the ~1000 hand-written call sites were migrated; the kind enums are emitted by the codegen, so re-running cargo run --bin generate-syntax reproduces them.
• Verified: cargo build --workspace --all-targets clean, clippy -D warnings clean, rust_fmt.sh clean, and the parser / syntax / formatter / doc / semantic test suites pass (two parser golden files regenerated for the diagnostic-wording

[reviewable-StarkWare]

This change is 

[orizi]

• refactor(syntax): Group SyntaxKind and unify token/terminal kinds under LexemeKind. #10154 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[TomerStarkware]

@TomerStarkware reviewed 23 files and all commit messages, and made 2 comments.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on eytan-starkware and orizi).

---

crates/cairo-lang-syntax/src/node/kind.rs line 383 at r1 (raw file):

    }
    pub fn is_keyword_token(&self) -> bool {
        matches!(

if let SyntaxKind::Token(lexemeKind) && lexemeKind.is_keyword_token() {
true
} else {
false
}

---

crates/cairo-lang-syntax/src/node/kind.rs line 420 at r1 (raw file):

    }
    pub fn is_keyword_terminal(&self) -> bool {
        matches!(

f let SyntaxKind::Terminal(lexemeKind) && lexemeKind.is_keyword_token() {
true
} else {
false
}

[orizi]

@orizi made 2 comments.
Reviewable status: 21 of 23 files reviewed, 2 unresolved discussions (waiting on eytan-starkware and TomerStarkware).

---

crates/cairo-lang-syntax/src/node/kind.rs line 383 at r1 (raw file):

Previously, TomerStarkware wrote…

if let SyntaxKind::Token(lexemeKind) && lexemeKind.is_keyword_token() {
true
} else {
false
}

Done.

---

crates/cairo-lang-syntax/src/node/kind.rs line 420 at r1 (raw file):

Previously, TomerStarkware wrote…

f let SyntaxKind::Terminal(lexemeKind) && lexemeKind.is_keyword_token() {
true
} else {
false
}

Done.

[TomerStarkware]

@TomerStarkware reviewed 8 files and all commit messages, made 1 comment, and resolved 2 discussions.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on eytan-starkware).

[cursor]

PR Summary

Medium Risk
Wide mechanical refactor across lexer, formatter, doc, and parser kind matching; low functional intent but high blast radius if any SyntaxKind pattern was missed.

Overview
Restructures how syntax node kinds are represented so lexical tokens and their terminal wrappers share one LexemeKind, with trivia and missing nodes split into TriviaKind and MissingKind instead of hundreds of flat Token* / Terminal* variants.

Call sites across doc, formatter, and parser now pattern-match SyntaxKind::Token(LexemeKind::…), SyntaxKind::Terminal(LexemeKind::…), SyntaxKind::TriviaToken(TriviaKind::…), and SyntaxKind::Missing(MissingKind::…) (e.g. ExprMissing → Missing(MissingKind::Expr)). Operator precedence and semicolon-skipping logic unwrap Terminal(lexeme) before using LexemeKind.

The lexer drops its private TokenKind enum and token_kind_to_terminal_syntax_kind; LexerTerminal carries LexemeKind directly. Parser diagnostics (MissingToken, inline-macro brackets, consecutive operators) use LexemeKind for display strings. colored_printer routes token coloring through an exhaustive set_lexeme_color over LexemeKind instead of a flat SyntaxKind match that could panic on unlisted kinds.

Intended behavior is unchanged except missing-token diagnostics may read Identifier instead of TerminalIdentifier.

^{Reviewed by Cursor Bugbot for commit 99f72de. Bugbot is set up for automated code reviews on this repo. Configure here.}

[eytan-starkware]

IMO the whole KindClassifier classify/path machinery is over-engineered — it reconstructs by name-string matching (the backs_terminal probe of format!("Terminal{x}"), classify() over 4 name lists) information cairo_spec already knows structurally: the lexeme tokens are exactly the add_token_and_terminal calls, and the trivia tokens are exactly the standalone add_token ones.

Can we drop it and rely on the spec instead? Concretely: an is_trivia flag on NodeKind::Token + an add_trivia_token builder (note the token-side is_keyword is write-only — keywords already come from the terminals — so it can just be replaced), and a single name→SyntaxKind-path map built in one pass over NodeKind: Terminal → Terminal(LexemeKind), Token → Token(LexemeKind)/TriviaToken(TriviaKind), enum missing variants → Missing(MissingKind), with TokenMissing special-cased by name; anything not in the map stays a flat variant. We checked locally — this drops ~100 lines from the generator with byte-identical generated files.

[eytan-starkware]

@eytan-starkware reviewed 4 files and all commit messages, and made 1 comment.
Reviewable status: 20 of 25 files reviewed, 1 unresolved discussion (waiting on orizi and TomerStarkware).

---

crates/cairo-lang-syntax-codegen/src/generator.rs line 87 at r3 (raw file):

        .filter(|n| !matches!(n.kind, NodeKind::Enum { .. }))
        .map(|n| n.name.as_str())
        .filter(|n| !paths.contains(n))

If paths were not strings this would have been without str compare. I am sure this is true for other places