Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
0259ed9
Host the Qwen3.8-Flash CUDA-graph decode loop in the Engine's static …
tianleiwu Oct 13, 2026
73e3189
Address PR #2728 review feedback on decode status propagation and spe…
tianleiwu Oct 17, 2026
55e3233
Support multi-draft S=2..4 cycles and ORT_QMOE_PREFILL_TOTAL_TOKENS i…
tianleiwu Oct 19, 2026
4ab2308
Adapt the Engine host to the reworked qwen38_flash_graph config and d…
tianleiwu Oct 31, 2026
1f4f4e4
Address PR #2728 review feedback on decode status after Drain
tianleiwu Oct 31, 2026
a87d6f5
Address PR #2728 review feedback on evaluated draft counts
tianleiwu Oct 31, 2026
9e16d24
Address PR #2728 review feedback on partial accept rounds
tianleiwu Oct 31, 2026
8d7e236
Address PR #2728 review feedback on the session-local expected prefil…
tianleiwu Oct 31, 2026
34a9245
Extract shared Engine backend factory, result contracts, and fixed-sl…
tianleiwu Oct 31, 2026
883635a
Fix MSVC C4858 in the engine backend factory tests
tianleiwu Oct 31, 2026
1acc0db
refactor(engine): remove the unused StageExecutorTokens path
tianleiwu Oct 31, 2026
b4ff8f4
fix(qwen38-executor): correct the teardown doc, cover the capacity an…
tianleiwu Oct 31, 2026
475d492
fix(engine): keep a displaced canceled request in the scheduler pool …
tianleiwu Nov 1, 2026
26447dc
fix(qwen38-executor): keep a failed status check fatal after the call…
tianleiwu Nov 1, 2026
09bc430
fix(engine): reject a declared unlimited request length that a finite…
tianleiwu Nov 1, 2026
207c4ad
fix(engine): address PR #2728 review feedback on FixedSlotScheduler, …
tianleiwu Nov 1, 2026
1786dd4
fix(engine): address review feedback on re-enrollment, deferred devic…
tianleiwu Nov 1, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions docs/paged_attention_engine.md
Original file line number Diff line number Diff line change
Expand Up @@ -2321,6 +2321,62 @@ tombstone.

Changes to shared types such as `Request`, `ScheduledRequests`, `ModelExecutor`, or `SimpleDecoder` should be checked against both paths. This document should be updated only where behavior is shared or where the dynamic path changes.

### Static single-slot backend that selects its own tokens

The `qwen38_flash_graph` model type (Qwen3.8-Flash CUDA-graph decode with a built-in MTP(1) loop) runs on the static path
through a `ModelExecutor` that chooses its own tokens, instead of attaching decoder state with logits. It is a single-slot
backend: fixed in-place KV, a recurrent-state bank ring and a device-side draft -> verify -> commit loop that cannot be
paged, batched or cropped to an arbitrary token. `CreateEngineBackend` (`src/engine/backend_factory.h`) and
`CreateQwen38GraphDependencies` (`src/engine/qwen38_graph_executor.h`) assemble it:

- `FixedSlotCacheManager` (`Qwen38GraphCacheManager`, `src/engine/fixed_slot_scheduler.h`) holds one row. `FixedSlotScheduler`
queues further Requests and recycles the slot when the resident turn is complete. `RewindToStartOfTurn` releases the slot; a
continued turn (`BeginTurn` on the resident Request) and the replay after a rewind re-ingest the whole retained history
(through the same chunked prefill as a new prompt), because the look-ahead cycle and the bank ring cannot be rolled back.
There is no prefix cache.
- `Qwen38GraphExecutor::SelectsOwnTokens()` is true. One Engine step is one prompt chunk (the size the device asks for,
`Qwen38DecodeDevice::PrefillChunkTokens()`: `model.qwen38_graph.prefill.chunk` when set, otherwise 8192 tokens with the
dense prefill (`prefill.enabled`) and 2048 per step through the decode graphs, see `EffectiveQwen38PrefillChunkTokens`;
the decode-graph path needs an even chunk; a step that only advances the prompt delivers no event, so `CancelRequest`
stays responsive; the dense prefill state continues across the chunks and is handed over to the decode state in the step that
ingests the last one) or one whole cycle (with `fold.enabled`: one CUDA-graph launch). The executor tells the device the
prompt length of the turn (`Qwen38DecodeDevice::ExpectPrefillTokens`, after `Reset()` and before the first chunk); the
device hands it to the ORT QMoE kernels with every chunk of that turn and drops it at `Reset()`/`BeginDecode()`. The
executor itself stores the expected length on the device instance rather than writing environment variables directly; the
production device then publishes `ORT_QMOE_PREFILL_TOTAL_TOKENS` through `Qwen38OrtEnvScope` around each ORT `Run` under a
process-wide mutex (isolating participating Qwen runs from each other, though unrelated concurrent ORT sessions in the same
process can still observe the process environment; see `src/models/qwen38_ort_env.h`). The
executor declares its step plan via `ScheduledRequests::BeginBackendStep` and stages owned `BackendRowResult` records
(`PrefillProgress` and/or `SelectedTokenBlock`) via `ScheduledRequests::StageBackendRow`; `GenerateNextTokens` validates the
batch before mutating any Request, commits candidates through `Request::CommitExternalTokens` (observing stop strings, EOS,
turn limit, and context limit in order), and records `TokenCommitAck` counts (consumed, appended, and staged for events)
settled through `TokenProducingSession` before events are published. Stop strings are therefore accepted on this static path;
every other static executor still rejects them.
- The next cycle is enqueued before a step returns, so the GPU runs it while the Engine delivers the previous tokens. A stop
string can only be known after delivery, so one cycle may be wasted; the next request, a rewind and Engine teardown wait for
it with `Qwen38DecodeDevice::Drain()`. The record of such a cycle is never consumed, so its status is read from the device
instead: `CheckDecodeStatus()` right after `Drain()` reports the error bits accumulated by every cycle since the last
`Reset()`, also when a record was left unconsumed. The executor always runs `Drain()` and then `CheckDecodeStatus()` before
`Reset()` (which clears the bits): a cancellation or stop-string match that leaves a cycle behind therefore still fails the
next request or the rewind when that cycle went wrong. At Engine teardown the executor drains the device and reads its status
too, but nobody is left to report an error to, so a failure poisons the device context instead (the next Engine over the
model fails). A fatal execution failure poisons the mutable device context across Engine leases until the model is reloaded.
- `ModelExecutor::ValidateTurn` and `BackendCapabilities` reject, before any Request mutation, what the device loop cannot do:
sampling, repetition penalty, no-repeat n-grams, `min_generated_tokens`, seeds and guidance. `MaxDraftTokensPerStep()` is
zero (the caller cannot attach drafts). `GetCapabilities()` reports one row and `max_request_length` = KV capacity - 4,
clamped to `model.context_length` when that is configured, so it is the smaller of the two.
- `GetSpeculativeStats()` includes the executor's counters (`ModelExecutor::AddSpeculativeStats`): one round per cycle, one
or more proposed drafts per round (the fold verifies 1..3), accepted drafts, bonus and correction tokens, the
acceptance-length histogram. The counters describe what the Engine consumed, as `Engine::RecordSpeculativeCommit` does for
sampled MTP: a turn that ends inside a cycle (turn or session limit, stop string, EOS) only accepts the drafts it delivered,
`draft_tokens_evaluated` counts the draft positions examined up to the terminal token (every accepted draft consumed, an EOS
draft included, plus the rejected position only when the turn got as far as the correction token), so `acceptance_rate` is not
diluted by drafts after the end of the turn. Rounds are classified like `Engine::RecordSpeculativeCommit` does, by the number
of accepted drafts that were delivered: none is a zero-accept round, all of the proposed ones a full-accept round and anything
in between a partial-accept round (`partial_accept_rounds`, counted whether or not a correction token was delivered).
- The model's device state is leased to one Engine at a time (`Qwen38GraphModel::AcquireDevice`); a second Engine over the same
model fails with a clear error until the first one is destroyed. The model has no Generator API support.

## Experimental low-level public API shape

The language bindings currently expose the same basic low-level loop. Production hosts are expected to wrap this surface or use its replacement rather than expose it as their stable API:
Expand Down
Loading
Loading