Repository navigation
Isolate MTP head metadata and decoder configuration - #2708
Draft
Tianlei Wu (tianleiwu) wants to merge 3 commits into
Draft
Tianlei Wu (tianleiwu) wants to merge 3 commits into
Tianlei Wu (tianleiwu) wants to merge 3 commits into
Conversation
3 tasks done
Split model builders, loaders, quantization, export tooling, builder tests, and builder documentation from kvaishnavi/qwen-38-flash (bf8e32c).
Stack runtime configuration, build integration, Engine and generator execution, Engram and hybrid/indexer state management, runtime tests, and runtime documentation on kvaishnavi/qwen-38-flash-create. Preserve committed changes from bf8e32c.
Keep the MTP head decoder's configuration independent from the target decoder: clear the target-only selected-logits input and state-update capacity when projecting the head config, and resolve the disabled indexer capacity from the head session's actual metadata instead of inheriting it from the target. A target-only setting can no longer leak into the head and produce mis-sized graph inputs when target and head differ. Adds ModelStateManifest::ResolveDisabledIndexerCapacity (rejects malformed or mismatched metadata without mutating state), unit tests and documentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Tianlei Wu (tianleiwu)
force-pushed
the
tlwu/20261012/ga01-head-metadata
branch
from
October 12, 2026 23:09
7c2240e to
61dafbd
Compare
Tianlei Wu (tianleiwu)
changed the base branch from
tlwu/20261012/qmoe-p0-base
to
kvaishnavi/qwen-38-flash-run
October 12, 2026 23:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Keeps the MTP head decoder's configuration and state metadata independent from the target decoder, and resolves the disabled-indexer capacity from the head session's actual metadata instead of inheriting it from the target. This prevents a target-only setting (selected-logits indices, compact-state/indexer capacity) from leaking into the head and producing mis-sized graph inputs when target and head differ.
Summary of Changes
Head/target isolation
src/config.cppCreateMtpDecoderConfigclears the target-onlylogits_indicesinput and always resets the head'sstate_update_capacityto 0.src/models/model.cppsrc/models/model_state_manifest.{h,cpp}ResolveDisabledIndexerCapacity, which reads the capacity from actual head session metadata and rejects malformed or mismatched metadata without mutating state.Tests and docs
test/cpp/engine/model_state_manifest_tests.cpptest/cpp/engine/mtp_config_tests.cpplogits_indicesand zero state-update capacity.docs/paged_attention_engine.mdIntentionally excluded
Offload-specific clearing, lookup/hybrid policy, capture/debug instrumentation and exporter changes are not included.
Testing
-DUSE_CUDA=OFF -DENABLE_PYTHON=OFF) on top of Add build and runtime support for Qwen-3.8 Flash Next #2714's head: 917 pass, 0 fail, 3 skipped vs 913 pass / 0 fail on the unmodified Add build and runtime support for Qwen-3.8 Flash Next #2714 head (+4 new tests).src/config.cppchange failsProjectsOneLayerFixedIndexerStateForTheHeadandResolvesDisabledHeadIndexerCapacityIndependentOfTarget; makingResolveDisabledIndexerCapacitya no-op fails both manifest rejection tests and the MTP resolve test; resolving in place instead of on a copy fails the no-mutation test.EnabledIndexerCapacityMismatchRemainsRejectedis a regression guard that passes either way, and themodel.cppcall site has no unit test because it needs a real session.git clang-format(20.1.8, the pinned version) andgit diff --checkare clean. Not run: CUDA, Python, guidance and telemetry surfaces.Motivation and Context
Found while qualifying head-only CUDA graph capture for Qwen MTP: a head that shares the target's selected-logits/indexer configuration can be planned with the wrong capacity. Making the head's own session metadata the source of truth removes that dependency. Rebased from the earlier
qmoe-p0-baselineage onto #2714; #2714 removedengine.max_graph_query_tokens_per_request, so the assertions on it were dropped.Checklist