GBNF / constrained decoding for LLM generate()? #1377

Summary

We're running LFM2.5-VL-1.6B on-device via react-native-executorch 0.9.3 (iOS / MLX). We'd like to confirm that RNE does not support GBNF (or any logits-level constrained decoding) on LLMModule.generate / forward, and ask whether that's something you'd consider adding.

If we missed an existing API, a pointer would be enough.

What we looked at

JS API (0.9.3)

• LLMModule.generate(messages, tools?) has no grammar / GBNF argument.
• GenerationConfig only exposes sampling knobs: temperature, topP / topp, minP, repetitionPenalty, plus token-batch timing. No grammar / schema / allowed-token set.

Native sampler

• Decode goes through TextDecoderRunner::logits_to_token → Sampler::sample.
• The sampler applies argmax or temperature + top-k + top-p + min-p + repetition penalty. We didn't find a grammar FSM, token mask from a GBNF, or a user logits processor.

"Structured output" in the docs

• getStructuredOutputPrompt / DEFAULT_STRUCTURED_OUTPUT_PROMPT inject schema instructions into the prompt.
• fixAndValidateStructuredOutput parses after generation.
• That's prompt + validate, not constrained decoding — the model can still emit anything.

Unrelated "grammar" in the repo

• The privacy-filter pipeline has a BIOES Viterbi grammar. That's token-classification, not LLM GBNF.

We also grepped open issues for GBNF / constrained decoding / logits processor and didn't find an existing thread. Happy to close this as a duplicate if one exists.

Why we care

We use GBNF on llama.cpp / llama.rn for small closed outputs, e.g. root ::= "Yes" | "No" or a short enum. On RNE we can't attach that to generate(), so we fall back to greedy decode + parsing (and first-token logprobs where we have them). That works until the model starts with "The …" instead of a label.

A logits mask / GBNF (llama.cpp-compatible), a JSON-schema sampler, or a per-step allowed-token callback would let us constrain the decode itself instead of hoping the prompt is enough.

Questions

1. Is the above correct for 0.9.x, and still true on the rewrite / 0.10 (#1208)?
2. If yes: would you consider constrained decoding on generate / forward?
3. If this is already possible through ExecuTorch LLM config / a runner hook we missed, please point us at it.

Happy to share a more precise sampler call-stack or a tiny repro if useful.