[Tianlei Wu (tianleiwu)]

Stacked on #2727 (base tlwu/20261013/ga-b3-qwen38-graph-model; the diff shows only this change). Position 3 of the series described in #2726. Retarget to main once #2727 merges.

Description

Serves the qwen38_flash_graph model type (#2727) through the Engine's existing static path. The CUDA-graph decode loop (the MTP draft -> verify -> commit cycle, the folded cycle, the dense fast prefill and the shared weight pool) stays on the device; the Engine keeps ownership of requests, turns, cancellation, stop conditions and speculative statistics. It is a single-slot backend: one resident request, greedy decoding, no prefix cache; further requests queue behind it in the static batch scheduler. Multi-slot hosting comes with ragged batching later in the series.

Multi-draft cycles of 2 to 4 tokens (S=2..4) are reported through Qwen38StepResult::tokens and the speculative statistics. The prompt length of a turn is handed to the device with Qwen38DecodeDevice::ExpectPrefillTokens (after the device Reset(), before the first prompt chunk), so QMoE memory lending can size the prefill budget from state that belongs to that device; the executor itself neither reads nor writes the process environment (ORT_QMOE_PREFILL_TOTAL_TOKENS is published by the device only around a single run). The prompt chunk size comes from EffectiveQwen38PrefillChunkTokens of the reworked qwen38_flash_graph configuration (#2727).

Summary of Changes

File	Change
src/engine/qwen38_graph_executor.{h,cpp}	Qwen38GraphCacheManager (static-cache contract for one row), Qwen38GraphExecutor (a ModelExecutor that selects its own tokens: one Engine step is one prompt chunk of the size the device asks for, or one whole decode cycle) and Qwen38GraphSession (platform- and CUDA-independent logic between the executor and a Qwen38DecodeDevice; checks the device error bits when a turn ends; continuation and rewind re-ingest the history through the same chunked prefill; the speculative statistics count the drafts that were actually evaluated and classify rounds by the drafts they delivered, as Engine::RecordSpeculativeCommit does).
src/engine/engine.cpp, model_executor.h, cache_manager.h	Hooks, all defaulting to the previous behaviour for other models: turn validation before any Request mutation (sampling, repetition penalty, no-repeat n-grams, minimum token counts, seeds, guidance and oversized sessions are rejected with a clear error), speculative-statistics reporting, static batch and length limits, and CreateDependencies returning early for this model type before any IndexShare/MTP setup.
src/engine/request.{h,cpp}, scheduled_requests.{h,cpp}	Request::CommitExternalTokens applies EOS, length limits and stop strings per committed token (precedence: stop string, EOS, turn limit, context limit; later tokens of a cycle are dropped); ScheduledRequests stages the executor's tokens. Stop strings are accepted on this static path only.
docs/paged_attention_engine.md	New section on the static single-slot backend that selects its own tokens: chunking, ExpectPrefillTokens, the Drain() + CheckDecodeStatus() contract and the speculative statistics.
test/cpp/engine/qwen38_graph_executor_tests.cpp, qwen38_graph_test_doubles.h	59 tests (plus one disabled benchmark) against a scripted Qwen38DecodeDevice and the real Engine: dense chunk streaming (any size, shorter than or exactly the chunk, hand-off exactly once and only after the last chunk), hand-off skipped when the first token ends the turn, cancellation between chunks and during decode, stop strings, EOS, continuation and rewind, all-rejected and all-accepted cycles, device errors (including failures that only show after Drain()), capability errors, evaluated-draft and round counts of truncated rounds, and the expected prefill total of two Engines on two owner threads. The scripted device completes cycles asynchronously like the real one (Drain() leaves the record in flight, CheckDecodeStatus() reports the accumulated status bits, Reset() clears them).

The dynamic/paged Engine path and the MTP path added by #2714 are not changed: the new Request/ScheduledRequests entry points are only reached from this executor, which never attaches drafts (MaxDraftTokensPerStep() == 0).

Review feedback incorporated

• Qwen38GraphSession::Release() and Begin() propagate an unchecked device decode status error (and capture a deferred teardown error), so a turn that ends through a stop string, cancellation, rewind or replacement still checks the device validation bits before Reset().
• The committed RequestStepResult is fed back to Qwen38GraphSession::OnStepCommitted() after Request::CommitExternalTokens, so tokens_emitted, tokens_discarded and the accepted and bonus counts reflect the tokens that were actually committed. Tests cover EOS, stop-string and canceled turns.
• A device failure that only becomes visible after Drain() (the look-ahead cycle that is completed when a turn is canceled, ends on a stop string or is rewound) fails the request instead of being cleared by the next Reset(). Every such path runs Drain() and then CheckDecodeStatus(), which on the reworked device contract (Add the qwen38_flash_graph model type and its device decode engine #2727) reads the status bits accumulated since the last Reset() even when a finished cycle record is left unconsumed, so no executor change was needed; the scripted device now behaves like the real one and AFailureOfTheLookAheadCycleVisibleOnlyAfterDrain* / AFailureOfTheFirstCycleVisibleOnlyAfterDrainSurfacesAfterACancellation cover it.
• The evaluated-draft count no longer over-counts a truncated round: it is the accepted-prefix positions the turn consumed plus the rejected position only when the round reached it (three accepted drafts with room for one token: 1 evaluated and 1 accepted, not 2 evaluated). Turn and session limits, stop strings and EOS never add a position beyond the boundary.
• Rounds are classified as zero, partial or full accept by the number of drafts the turn took from the round, partial_accept_rounds is counted for every partial round, and AddStats merges it.
• The expected prefill total is state of the device (ExpectPrefillTokens), not of the process: the executor no longer calls setenv. EnginesOnTwoOwnerThreadsKeepTheirOwnExpectedPrefillTotals runs two Engines on two owner threads with their own scripted devices and checks that each device sees its own total in every chunk of a turn (first turn, continued turn, canceled turn, following request) and that ORT_QMOE_PREFILL_TOTAL_TOKENS is untouched.

Testing

• CUDA build (-DUSE_CUDA=ON -DENABLE_QWEN38_PLANE=ON, CUDA 13.0, H200/sm_90) on top of the reworked Add the qwen38_flash_graph model type and its device decode engine #2727 (85595784): engine_unit_tests --gtest_filter='*Qwen38*' 135 passed, 0 failed. The full binary runs 1249 tests: 1209 passed, 22 skipped, 18 failed, and the 18 are exactly the failures of the unmodified base in this environment (the Cuda* provider-bridge tests need GetProvider from the in-tree CUDA EP, which the rc5 plugin EP used here does not export; the three ModelPackage.* tests (TokenizerResolvesThroughSharedAsset, ExternalInitializersFolderResolvesThroughSharedAsset, FlatDirectoryRejectsSha256Reference) fail identically on unmodified main and are unrelated to this change).
• The sources of this PR compile as C++17 with -DUSE_CXX17=1 (the Android configuration) and as C++20 with -Wall -Wextra without diagnostics.
• GPU qualification of this code through og.Engine (original head of this PR, H200, v12 package): token identity against the reference runner 6/6 (EOS mode) and 6/6 (fixed length) with equal cycle and accept counts, MTP stream equal to the no-MTP stream 6/6, and engine_behaviours.py 27/27 in the bit-exact (--prefill steps) and the fast-prefill modes. The tip of the whole stack was re-qualified end to end on the rc5 package (see the final PR of the runtime series, [Qwen38] Adopt the SpeculativePrefixCommit contract of ORT #33474 and fix the conv-state replay orientation #2824).
• Not run: Windows/macOS, multi-GPU, sampling (rejected by design).

Motivation and Context

The Generator-hosted version of this loop could not share the Engine's request lifecycle (turns, cancellation, stop strings, statistics, queueing), which is what the serving stack uses. Hosting it as a static-path executor reuses all of that while keeping the single-slot device state (fixed in-place KV, recurrent-state bank ring) that cannot be paged. It is an alternative backend next to the paged Engine MTP path of #2714, not a replacement.

Limits of this PR: greedy only; one resident request; no prefix cache and no caller drafts; a continued turn or the replay after RewindToStartOfTurn re-ingests the whole history (0.3 s at 2k, 3.2 s at 32k with the fast prefill); sessions up to kv_capacity - 4 tokens; Linux + CUDA only. The GPU gates need an exported package, the CUDA EP and the contrib operators, which CI does not have, so CI covers the scripted-device tests only.

Checklist

• Tests added/updated
• No breaking changes (hooks default to the previous behaviour)
• Documentation updated

[Tianlei Wu (tianleiwu)]

Rebased onto the new head of #2727 and force-pushed (4719a3f05 -> e864af904). The description was rewritten to match the re-cut series; the series map is in #2726. The head includes a whitespace-only clang-format pass to match the CI's clang-format 20. Re-based again onto the amended #2726 (Windows rooted-path fix); the change of this PR is the same.

[Copilot]

🟡 Changes recommended

Pending look-ahead cycles can silently discard device errors after a stop string or cancellation, while multi-draft acceptance statistics also remain incorrect.

3 open issues

• Retire pending cycles before checking device status · New
• Exclude drafts beyond the terminal boundary from acceptance metrics · New
• Count and merge partial-accept round classifications · New

What changed in this PR

Hosts the qwen38_flash_graph device backend introduced in #2727 through the Engine’s static path, integrating GPU decode cycles with request lifecycle management.

Changes:

• Adds a single-slot executor and cache manager with chunked prefill, continuation, and rewind support.
• Adds token-wise commit handling for stop conditions and hooks for capability validation and speculative statistics.
• Documents the backend and adds scripted-device tests for lifecycle, decoding, and error handling.

File	Description
test/​cpp/​engine/​qwen38_graph_test_doubles.h	Adds a scripted decode device for Engine tests.
test/​cpp/​engine/​qwen38_graph_executor_tests.cpp	Tests decoding, lifecycle, limits, and device failures.
src/​engine/​scheduled_requests.h	Declares executor-token staging.
src/​engine/​scheduled_requests.cpp	Routes staged tokens through request commit handling.
src/​engine/​request.h	Declares external-token commit support.
src/​engine/​request.cpp	Applies stop conditions to executor-selected tokens.
src/​engine/​qwen38_graph_executor.h	Defines the session, executor, and single-slot cache manager.
src/​engine/​qwen38_graph_executor.cpp	Implements device orchestration and speculative accounting.
src/​engine/​model_executor.h	Adds token-selection, validation, and statistics hooks.
src/​engine/​engine.cpp	Integrates backend creation, commit notifications, and capabilities.
src/​engine/​cache_manager.h	Adds static batch and session-length capability hooks.
docs/​paged_attention_engine.md	Documents the backend’s behavior and restrictions.

_{Review effort: Balanced}

[Akshay Sonawane (apsonawane)]

Reviewed all 12 changed files at e864af9 plus the production device, request, cache and stats contracts. One new cross-Engine environment-state defect is recorded inline. The three existing unresolved threads on deferred status, evaluated-draft counts after truncation, and partial-accept accounting remain valid; I did not create duplicate findings. The scripted device's Drain behavior should not be taken as evidence that the production status path is fixed. No repository build, GPU execution or actual concurrent-Engine test was performed; this is source/control-flow review, not approval.