Add build and runtime support for Qwen-3.8 Flash Next by kunal-vaishnavi · Pull Request #2714 · microsoft/onnxruntime-genai

kunal-vaishnavi · 2026-10-12T20:46:48Z

Description

Adds build and runtime execution support for the Qwen-3.8 Flash Next graphs exported by #2713. The updated target branch has been merged, not cherry-picked.

Configuration/model integration for Engram, sparse-indexer state, PLE, and MTP.
Full T/H/W rotary-history retention across Generator decoding, reset/rewind, and main-to-MTP handoff; stable history buffers and packed-text history bindings match the builder interface.
Legacy recurrent-window compatibility plus validated compact replay, snapshot baselines, and accepted-prefix rollback.
IndexShare graph-derived capacity clamping, recoverable merge-status fallback, budgeted persistent stage scratch, stable layout keys, and asynchronous failure-lifetime fences.
Engram batch-cache correctness, independent token/Engram embedding inputs, explicit child-provider settings, and stable per-length capture buffers.
Replay dtype/capacity validation and ORT-matching CPU/CUDA arithmetic.
Removal of experimental decoding/debug/chunking switches; compatible examples cleanup and standalone MSVC security defaults.

Validation

Linux CUDA Debug core/provider, Python wheel, and C examples build successfully. 1,040 C++ Engine/component tests passed, 21 skipped; 159 CPU/CUDA Python integration/build-helper/example tests passed, 1 skipped. Coverage includes actual coordinate-history retention and MTP handoff, legacy windows, sampling/rejection baselines, IndexShare failure/cancellation/shrinking batches, CUDA replay parity, and captured dual-embedding decoding. Builder tests are validated in #2713.

Provider add-ons must be rebuilt against device-interface version 13 because the configuration/position interface layout changed. The Engine retains its existing text-only restriction for packed multimodal coordinates; multimodal Generator and exported graph coordinate handling are no longer blocked. Validation uses focused numerical/synthetic fixtures rather than claiming full-checkpoint performance results.

Motivation + Context

Qwen-3.8 Flash Next combines sparse attention, recurrent layers, Engram, and multi-token prediction. The runtime must coordinate these stateful components and correctly commit or restore them during speculative execution.

This PR targets kvaishnavi/qwen-38-flash-create. The full integration branch including experiments is kvaishnavi/qwen-38-flash.

Split model builders, loaders, quantization, export tooling, builder tests, and builder documentation from kvaishnavi/qwen-38-flash (bf8e32c).

Stack runtime configuration, build integration, Engine and generator execution, Engram and hybrid/indexer state management, runtime tests, and runtime documentation on kvaishnavi/qwen-38-flash-create. Preserve committed changes from bf8e32c.

Tianlei Wu (tianleiwu)

Review summary

Thanks for splitting this out; the runtime design is coherent and carefully layered: MTP drafting stays an optional acceleration inside a mandatory target transaction, the shared past/present indexer buffer is made rollback-safe with double-buffered lengths, the CPU and CUDA replay paths agree on the cases traced by hand, the persistent graph buffers are described once and priced into the reserved-memory budget, the device-interface version is bumped, and the EOS-floor masking has dedicated tests.

I built the PR head CPU-only and ran the engine tests: engine_unit_tests 913 passed / 3 skipped (CUDA and real-model), unit_tests 252 passed, ctest 9/9. Not run: CUDA code, the Python engine tests (need CUDA and ORT ops), and any Windows/macOS build. The CUDA-side comments below come from reading the code.

Changes recommended before merge (inline comments marked High):

Engine::PrepareMtpStep turns two data/config-dependent conditions (exported IndexShare draft capacity < request budget, non-zero merge status) into std::logic_error. Those are rethrown and mark the engine unhealthy, instead of taking the documented "fall back to target-only decoding" path.
RecurrentState now rejects rank-5 windowed state. state_window exports used to load and crop; docs/speculative_decoding.md still documents that layout.
cmake/msvc_security.cmake silently loses /Qspectre in the standalone src/python and src/java builds, because ENABLE_MSVC_SPECTRE is only defined in options.cmake.
build.py uses shutil.rmtree(onexc=...), which needs Python 3.12; the Windows workflows pin 3.11.

Cross-cutting concerns

Behaviour change for existing models via MtpGenerator: the main-model SnapshotState() is now unconditional, the N=1 reject path crops the verify state instead of re-decoding, and new ORTGENAI_MTP_* environment switches are read from the environment without validation or docs. The default path of existing Qwen 3.5/3.6 MTP users is affected; please call this out in the description or gate it.
Shared multimodal path: the new MultiModalPipelineState::RewindTo and the hard-coded adaptive_chunking thresholds apply to every VLM.
Test gaps: no C++ test drives PrepareMtpStep with IndexShare (capacity mismatch, merge failure, partial accept + reseed, cancel/rewind), and there are no direct tests for EngramState, IndexerCache, PleState or the Qwen4Exp preprocessing.
Hot path: CpuEngram is rebuilt every decode step with one ORT call per request, and the IndexShare chain allocates per step; both work against the CUDA-graph decode goal.

Hygiene: stray blank-line-only edits in src/generator/generators.cpp and mtp_generator.cpp; about 38 lines of CRLF/LF churn in test/CMakeLists.txt; an unrelated wheel-packaging exclude for models/gemma* in src/python/CMakeLists.txt; the adaptive_chunking addition is a legitimate fix (python.cpp already calls it) but is not mentioned in the description; std::getenv is read on the per-step rollback path in recurrent_state.cpp (read once instead).

Verdict: COMMENT. I would want the four High items above addressed (or explicitly deferred) before merge; the rest are suggestions.

Baiju Meswani (baijumeswani)

Please address the issues below. Some settings can return the wrong results, fail during generation, or silently ignore the requested device.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Move paged attention graph and runtime configuration helpers out of the core builder. Separate Qwen3.5 and Qwen3.8 graph expansions from model orchestration, and add dispatch regression coverage and architecture documentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove the package conversion script and its dedicated test. Document re-exporting through Model Builder instead, preserving built-in IndexShare export support. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Separate vision, Engram, MTP and standard-op expansions; restore explicit per-op CUDA layouts and preserve provider behavior. Include removed sparse-attention session options from the sibling clone. Keep index sharing intrinsic to paged main/MTP graphs, validate export limits, and reject unsupported multimodal rotary history rather than emitting incorrect graphs. Validation: 2107 builder tests passed, 18 skipped; changed tests/example and standalone modules pass Ruff; Python compilation passes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Bind the shared Engram conversion helper directly instead of inserting a constructor-free mixin ahead of initialized builder bases. Preserve the same emission behavior and make projection return paths uniformly explicit. Validation: 320 affected architecture, Engram, native-weight and numerical regression tests passed; Ruff and formatting passed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Fix the cross-platform Python CI failure outside the builder test directory by initializing the synthetic loader packing configuration. Document standalone component loaders and the explicit, host-independent NVFP4 layout contract. Validation: 2288 builder and related model-loader tests passed, 18 skipped; Ruff, formatting, Python compilation and diff checks pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Keep companion graph construction assertions on all providers, but skip numerical execution when the pinned ORT lacks the required contrib operator. Probe independent minimal graphs so shape and other runtime errors are not hidden. Add tests for missing-operator detection and propagation of unexpected inference errors. Validation: 2292 builder and related model-loader tests passed, 18 skipped; Ruff and formatting passed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ash-create

Apply Q/K normalization and current-step MRoPE before the sparse executor, recompose per-request indexer rotary tables from explicit full T/H/W history, and retain the matching input on multimodal MTP heads. Remove the temporary export rejection. Also emit full-width QSA split-half rotary tables instead of passing attention half-width caches. Document the history contract for the stacked runtime integration. Validation: 2301 builder and related loader tests passed, 18 skipped, including dense/packed numerical MRoPE, history recomposition and operator-ABI tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Preserve the runtime example model-family coverage and target-side session/initializer mappings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Emit architecture compression metadata for runtime rollback and consistently use the first EOS ID when a checkpoint supplies multiple EOS IDs. Reject missing, nonintegral, or negative Engram IDs before export. Validation: 2310 builder and related model-loader tests passed, 18 skipped. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

… into kvaishnavi/qwen-38-flash-run

…ettings Use ONNX IR use-def editing for IndexShare while retaining external references and metadata. Group Qwen sparse-indexer, PLE, and hyper-connection settings in documented attribute dictionaries. Honor resolved paged state capacity, preserve boolean fusion CLI aliases, and reconcile configurable fusion/allocator documentation. Validation: 2326 builder and related loader tests passed, 18 skipped; 54 focused tests passed with minimum supported onnx_ir 0.1.3. Changed-code Ruff checks pass apart from unchanged excluded builder.py baseline findings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

… into kvaishnavi/qwen-38-flash-run

Retain full T/H/W prompt histories across decoding, rewind and main/MTP execution. Preserve legacy recurrent windows, compact replay baselines, host-control graph variants and per-length stable embedding buffers. Clamp IndexShare capacity, recover from merge status, budget persistent scratch and fence failing asynchronous work. Fix replay bounds/dtypes/ORT-exact arithmetic, Engram batch cache keys/providers and dual embedding inputs. Restore compatible build cleanup and standalone MSVC security defaults; remove experimental debug/chunking behavior. Validation: CUDA Debug core, provider, examples and wheel build; 1023 C++ Engine/component tests passed, 3 skipped; 120 CPU/CUDA Python integration/build tests passed, 1 skipped. Includes actual image-coordinate history, CUDA graph dual-input, IndexShare failure/cancellation, and replay parity regressions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Annotate packed Q/K norm intermediates with token/head axes instead of dense batch symbols. Assert numerical MRoPE metadata and execute dense/quantized IndexShare projection fixtures with zero and nonzero selected rows while retaining input/output metadata. Validation: 2334 builder and related loader tests passed, 18 skipped; focused IR regressions passed and minimum supported IR compatibility remains covered. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove the legacy fuse_mlp_gate_up control and structured target optimization field; reject stale settings explicitly and remove their help/config examples. Enable target fusion by default while retaining separate projections for adapted/prequantized/nonfloat weights, independent exclusions, exact node overrides or incompatible precision placement. Apply the same eligibility to Qwen shared experts and retain mixed-precision entries for both possible layouts. DFlash2 independent fusion policy is unchanged. Validation: 2479 full builder/quantized-loader tests passed, 21 skipped; actual default fused ONNX execution matches unfused PyTorch and emits two MatMuls plus Split. Added provider-wide default dispatch, fallback and removed-option regressions. Scoped Ruff and whitespace checks passed; baseline comparison confirms no new base/CLI builder lint findings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove fuse_qkv, disable_qkv_fusion, fuse_qk_norm_gqa and IndexShare-specific extra-option handling rather than maintaining aliases or rejection paths. Use automatic fusion eligibility and the existing max_draft_tokens budget. Remove target structured QKV controls and stale help/examples while preserving independent DFlash2 settings. Also eliminate special rejection handling for the previously removed target MLP fusion option as requested. Cover provider/LoRA/attention/RoPE compatibility without export toggles. Validation: 2471 full builder/quantized-loader tests passed, 21 skipped; scoped Ruff and whitespace checks passed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Return existing paged cache/state, varlen, rotary, attention and selected-row methods to their original positions relative to surviving methods on the PR base. Restore Qwen35 attention/state-group ordering from its original class and place paged MRoPE/init/config/LM-head helpers beside their related operations instead of at EOF. Function bodies and class statements remain AST-identical. Add grouping regressions. Validation: 412 targeted tests passed; scoped Ruff and whitespace checks passed; baseline ordering and AST equivalence verified. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Revert the WebGPU expansion file exactly to the PR base, including the original method name, comments and formatting. Restore the matching graph-capture binding in Model and adjust ownership/binding regressions to recognize the original override name. Validation: no WebGPU file diff against base; 86 tests passed, 15 skipped; scoped Ruff and whitespace checks passed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Migrate CLM/KEV graph construction, MTP IndexShare inspection/save, CUDA weight prepacking, KV-cache variants and CPU-embedding splitting to IR values, nodes, attributes and tensors. Use IR graph inspection for component packaging while retaining raw malformed external-data validation and the independent ONNX structural checker. Preserve metadata, lazy external tensor references, nested captures, public protobuf-facing tool boundaries and numerical export contracts. Add source-wide helper API regression coverage and targeted transformation/metadata tests. Validation: full builder/quantized-loader suite 2507 passed, 21 skipped on ONNX IR 1.0.0; all migrated paths 672 tests passed on minimum ONNX IR 0.1.3; scoped Ruff and diff whitespace checks passed, with no new base-builder lint diagnostics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…nal storage Remove persistent hf_load_config and use local HF configuration for nested decoder layer limits. Remove the MLP fusion attribute/eligibility method and combine compatible emitted gate/up MatMul or MatMulNBits graphs by default. Inline paged input/output bindings and MRoPE rank handling, restore original initialization placement and require named intermediate Reshape outputs. Preserve explicit host-independent MatMulNBits/QMoE packing controls and their documentation. Restore Standard normalization and fused-op decompositions under unsuffixed provider override names, reuse existing normalization APIs, and remove offset-RMSNorm/expansion-constant helpers. Save generated companion/drafter/tool graphs with one model-owned external data file while retaining runtime initializer sharing metadata; preserve small Constant attributes needed by ORT shape inference. Update regressions, docs and spacing per Model Builder guidance. Validation: full current-IR builder/quantized-loader suite 2573 passed, 21 skipped. Exact minimum IR 0.1.3 focused normalization/fusion/config tests 542 passed; external/model tests 143 passed plus extended tool/rollback suite 108 passed. Full minimum-version failures exactly match all 41 pre-change baseline failed/error cases. Changed-file lint has only existing base.py diagnostics; whitespace checks pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Document automatic optimizations and minimal configuration/state, appropriate builder/Standard/provider ownership, ONNX IR and existing helper reuse, explicit deployment packing, per-model external data, stable naming/placement/formatting and evidence-driven validation. Reconcile overlapping guidance and retain model-builder-scoped instruction discovery. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…nfrastructure Make CLM/KEV and composite support inherit Model and reuse graph/node/value/tensor/save infrastructure. Emit paged Qwen MTP index-sharing bindings, gated projection, merge and status directly during construction; remove serialized graph converters. Enable drafter fusions by default without attn_attrs or attention-metadata options. Export excluded DFlash embeddings natively from decoder weights and reuse decoder embedding tensors/serialized bytes through existing initializer adoption logic. Remove post-export split/KV-variant scripts; integration tests use native exports. Organize Qwen3.8 under builders/qwen3_8 with stable package reexports, reuse packed MatMul helpers and move generic fused-op wrappers into Model. Route Standard norms through layernorm dispatch and pass branch counts explicitly. Remove all fusion option schema/help, preserve initializer allocator control and broaden text_only help. Isolate raw ONNX source validation/checking in a component loader so builder_config has only IR-facing parsing. Preserve validation that public IR APIs cannot supply. Move model-specific docs under models/docs and enforce infrastructure/no-post-export-surgery instructions. Validation: 2525 builder/quantized-loader tests passed, 21 skipped; exact minimum ONNX IR 0.1.3 focused suite 795 passed; CPU/CUDA synthetic embedding runtime parity 2 passed, 3 opt-in real-model tests skipped. Scoped Ruff, Markdown/JSON/frontmatter/link and whitespace checks passed; base/CLI have no new lint findings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Qwen3.8-Flash-Next needs the runtime in microsoft/onnxruntime-genai#2714, which is stacked on #2713 and on main changes made after v0.17.0. Add the 35 main PRs those diffs need, #2713, #2714 and #2782 (IndexerCache fix on top of #2714). Drop 2705, superseded by #2672 which is now in the list. Exclude Markdown from the applied diffs and retry .diff fetches on Windows.

Remove Qwen38Engram and the decoder internal hashing/embedding-table path. Always bind standalone engram_embeddings; only engram.onnx stores the table. Extend Qwen4ExpEngramModel with paged VarlenNGramHashMapping internals while preserving the existing rank-two tokens/rank-three embeddings runtime ABI. Keep hidden-state-dependent projection/gate/conv processing inside the decoder. Forward standalone present-token state through the existing Engine input/output binding and capture speculative token-prefix snapshots using standard operators rather than a duplicated hash op. Validation: full builder/quantized-loader suite2559 passed,23 skipped; exact minimum ONNX IR0.1.3 focused376 passed,2 skipped. New numerical tests cover multi-request dense/paged equivalence, EOS and history, metadata/per-model external files, unequal/empty token snapshot inputs and accepted-prefix rollback. Two focused skips reflect existing ORT Varlen hash empty-request restriction. Scoped Ruff and whitespace passed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ayouts Remove varlen hashing adaptation, shape-processing helper and attention-layout constructor flag. Use the original NGramHashMapping/GatherBlockQuantized/optional Cast/Reshape graph for both dense and paged decoder exports; runtime request packing remains unchanged. Preserve standalone-only table ownership, token-history handoff and decoder speculative rollback logic. Update docs and regressions, including successful empty requests without varlen-kernel skips. Validation: standalone Engram source exactly matches pre-adaptation version; 292 focused tests passed; full builder/quantized-loader suite 2561 passed,21 skipped; scoped Ruff and whitespace passed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…regressions Name real producer outputs directly and share unchanged dense history input/output values without Identity nodes. Handle predeclared graph outputs and metadata through portable IR use-def updates. Make graph-only companion/composite/drafter constructors explicitly initialize Model to address CodeQL superclass findings. Split Qwen3.5 into text/model/mtp package modules with stable exports and inheritance. Discover nested Python packages automatically so Qwen packages are present in wheels; regression checks cover all model packages. Skip only genuinely unavailable MRotaryEmbedding on older ORT CI builds while preserving invalid-graph errors. Audit every loader for helper APIs and add malformed/nested source metadata tests; retain only the necessary raw ONNX parser/checker boundary documented in LOADERS.md. Validation: full builder/quantized-loader/component audit suite2585 passed,21 skipped; exact minimum ONNX IR0.1.3 focused653 passed; CPU/CUDA synthetic embedding parity2 passed,3 optional real-model skipped. Scoped Ruff/whitespace passed with no new base lint diagnostics. NuGet restore failure identified as external dependency infrastructure; fresh remote CI will validate the pushed head. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Qwen3.8-Flash-Next needs the runtime in microsoft/onnxruntime-genai#2714, which is stacked on #2713 and on main changes made after v0.17.0. Add the 35 main PRs those diffs need, #2713, #2714 and #2782 (IndexerCache fix on top of #2714). Drop 2705, superseded by #2672 which is now in the list. Exclude Markdown from the applied diffs and retry .diff fetches on Windows.

Use a single rooted shutdown handle, register its exit handler once, and dispose test-owned sequences, streams, images and audio deterministically. Exercise repeated test construction and GC while a generator is live. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ership Resolve the C# test harness overlap using main’s module initializer as the sole process-lifetime shutdown owner. Retain the live-generator GC regression and deterministic resource cleanup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

### Description Adds the model-builder foundation for Qwen-3.8 Flash Next: - Standalone text, vision, embedding, Engram, and MTP builders, with component-specific loaders and expansions. - Shared standard-ONNX expansions reused by provider adapters without overriding unrelated CUDA/WebGPU paths; existing dense CPU/DML/TRT-RTX LinearAttention remains supported. - Native FP8/NVFP4 checkpoint support and explicit, host-independent per-op weight packing controls. - Sparse-attention indexer, recurrent, PLE/Engram, and MTP graph/state metadata. - Index sharing intrinsic to paged main/MTP graphs, without a conversion tool or enable flag; ONNX IR graph editing preserves use-def relationships, metadata, and external weight references. - Canonical documented operator settings, consistent CLI/config documentation, and regression tests. ### Multimodal Rotary Contract The temporary multimodal export restriction has been removed. Current Q/K use all three T/H/W position streams before sparse attention. Request-specific indexer rotary tables are recomposed from the full `rotary_position_ids` history, including paged graphs and multimodal MTP heads. Text-only exports retain their existing interface. QSA uses full-width split-half rotary tables, not the attention half-width cache. Matching history retention, rewind, and main-to-MTP runtime support is implemented in the stacked runtime PR #2714. This PR targets main and defines graph construction; #2714 targets this branch and supplies execution support. ### Validation **2,505 builder and related model-loader tests passed, 21 skipped.** Tests cover dense/packed numerical MRoPE, historical spatial positions, frame isolation, Engram padding/scales, packing eligibility, state capacity, metadata/external-data preservation, and zero-row IndexShare projection reuse. Minimum supported ONNX IR compatibility was also checked. These are targeted numerical and synthetic regression checks, not a claim of full-checkpoint benchmark validation. The full integration branch including experiments is `kvaishnavi/qwen-38-flash`. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

… (#1335) ## Summary Build OGA v0.17.0 with the Qwen3.8-Flash-Next runtime: microsoft/onnxruntime-genai#2801 (the Generator-path runtime of #2714 plus the IndexerCache fix #2782, ported onto v0.17.0) and microsoft/onnxruntime-genai#2792 (onnxruntime-extensions pin with `Qwen3_5Tokenizer`). ## Related issue or design microsoft/onnxruntime-genai#2714, microsoft/onnxruntime-genai#2782, microsoft/onnxruntime-genai#2792, microsoft/onnxruntime-genai#2801 ## Why The pinned OGA rejects the Qwen3.8 `genai_config.json` (`Unknown value "rotary_position_ids"`). #2714 adds that runtime but is based on OGA main after v0.17.0, so its diff does not apply to the tag alone. #2801 carries only the Generator-path part of #2714 (no Engine or CUDA changes) on top of v0.17.0, so no other main PRs are needed. ## What - `OGA_PR_PATCHES` (Windows and Linux): `2685 2691 2705 2712 2792 2801`. - #2792 replaces 2763: it pins onnxruntime-extensions main plus microsoft/onnxruntime-extensions#1136 (the pre-tokenizer fix 2763 carried), which also brings microsoft/onnxruntime-extensions#1142 (`Qwen3_5Tokenizer`). 2763 now merges OGA main into its branch, so its diff no longer applies on v0.17.0. Linux gets the pin too. - #2801 is applied after #2685, which rewrites the same extensions pin line, so the pin stays in its own PR. - Exclude `*.md` from the applied diffs; the build does not read Markdown. - Windows: retry the `.diff` fetches (`-MaximumRetryCount 6`), as Linux already does with curl. ## Test plan - [x] The list applies cleanly on v0.17.0 with the CI `git apply --3way` command (locally) - [x] build-windows / build-linux green with this list (#1351) - [x] Qwen3.8-Flash-Next with the Windows `gpu-test-package`: image description correct; 1024-token prefill / decode on par with the earlier OGA build from #2714 plus its main dependencies - [ ] GPU tests show no regression on the existing OGA models ## Notes for reviewers The OGA cache key only contains PR numbers, so pushing new commits to #2792 or #2801 does not invalidate it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Retain Qwen runtime history and replay features with upstream graph capture and multimodal buffer changes. Bump the combined device interface ABI to 14. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…etadata Keep the preinstalled CUDA toolchain unchanged while avoiding stale NVIDIA metadata during the shared OS-only dependency step. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

kunal-vaishnavi · 2026-10-29T07:13:26Z

Addressed the three failing checks at runtime sync d719a7a in commits 8023cc7, 7a8321b, and 299bb22. Corrected C++ formatting (and simplified equivalent nested state-output/rank conditions). The CUDA failures were stale NVIDIA repository metadata returning 404 during the OS-only dnf dependency step. The shared installer now excludes cuda* repositories for that step; CUDA remains the version supplied by the existing base images, and package feeds are otherwise unchanged.

Verified on 299bb22: Clang Format Lint passed; both x64 CUDA Get Docker Image and ARM64 Build ARM64 CUDA manylinux image passed their previously failing provisioning steps; the full Linux CPU x64 workflow passed native compilation, C# tests, and tests. ARM64 CPU image provisioning also passed, covering the shared script on a non-CUDA image. CUDA compilation and other platform jobs are still running; no current failures are reported at this update. This is not a claim that all CI has finished.

kunal-vaishnavi added 2 commits October 12, 2026 20:30

Add Qwen-3.8 Flash model builder and export support

e638c1b

Split model builders, loaders, quantization, export tooling, builder tests, and builder documentation from kvaishnavi/qwen-38-flash (bf8e32c).

kunal-vaishnavi added this pull request to stack #2716 October 12, 2026 20:48

kunal-vaishnavi mentioned this pull request Oct 12, 2026

Support Qwen3.8-Flash-Next (Qwen4ExpForConditionalGeneration) in Model Builder #2598

Open

This was referenced Oct 12, 2026

Isolate MTP head metadata and decoder configuration #2708

Draft

Guard graph buffer packed-row and position dimensions #2709

Draft

Tianlei Wu (tianleiwu) reviewed Oct 12, 2026

View reviewed changes

Baiju Meswani (baijumeswani) reviewed Oct 16, 2026

View reviewed changes

kunal-vaishnavi and others added 17 commits October 19, 2026 01:21

Merge main into Qwen3.8 Flash model builder PR

94bbf2c

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove standalone IndexShare MTP conversion utility

878d396

Remove the package conversion script and its dedicated test. Document re-exporting through Model Builder instead, preserving built-in IndexShare export support. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge origin/main into Qwen3.8 builder revisions

21a1982

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into kvaishnavi/qwen-38-fl…

7eefcf2

…ash-create

Merge updated Qwen builder target into runtime PR

feecb19

Preserve the runtime example model-family coverage and target-side session/initializer mappings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge remote-tracking branch 'origin/kvaishnavi/qwen-38-flash-create'…

50a42f0

… into kvaishnavi/qwen-38-flash-run

Merge remote-tracking branch 'origin/kvaishnavi/qwen-38-flash-create'…

207c38d

… into kvaishnavi/qwen-38-flash-run

kunal-vaishnavi and others added 4 commits October 26, 2026 04:49

Wang, Zhenze (zz002) mentioned this pull request Oct 26, 2026

ci(oga): apply the Qwen3.8 runtime via microsoft/onnxruntime-genai#2801 ROCm/hip-ep#1335

Merged

3 of 4 tasks

kunal-vaishnavi and others added 4 commits October 26, 2026 22:24

kunal-vaishnavi and others added 7 commits October 27, 2026 08:08

Fix component builder method contract collisions from review

4b7db00

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Fix packed drafter WebGPU test feeds and Windows CUDA NuGet restore

c6a29ab

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Restore internal NuGet source for Windows CUDA C# tests

030db7a

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Normalize function spacing in modified Model Builder files

35a9d02

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Wang, Zhenze (zz002) mentioned this pull request Oct 28, 2026

[Do not merge] Qwen-3.8 Flash Next runtime on v0.17.0 #2801

Draft

kunal-vaishnavi and others added 2 commits October 28, 2026 08:24

Base automatically changed from kvaishnavi/qwen-38-flash-create to main October 29, 2026 00:43

kunal-vaishnavi and others added 5 commits October 29, 2026 05:54

Merge final Qwen builder changes into runtime branch

bf66609

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge current main into Qwen runtime PR

d719a7a

Retain Qwen runtime history and replay features with upstream graph capture and multimodal buffer changes. Bump the combined device interface ABI to 14. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Fix runtime formatting and isolate OS dependency installs from CUDA m…

8023cc7

…etadata Keep the preinstalled CUDA toolchain unchanged while avoiding stale NVIDIA metadata during the shared OS-only dependency step. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Allow OS dependency setup on images without a CUDA repository

7a8321b

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Simplify nested state condition formatting and test JSON literals

299bb22

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Navigation Menu

Repository navigation

Add build and runtime support for Qwen-3.8 Flash Next - #2714

Add build and runtime support for Qwen-3.8 Flash Next#2714
kunal-vaishnavi wants to merge 66 commits into
mainfrom
kvaishnavi/qwen-38-flash-run

kunal-vaishnavi commented Oct 12, 2026 •

edited

Loading

Uh oh!

Tianlei Wu (tianleiwu) left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Baiju Meswani (baijumeswani) left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kunal-vaishnavi commented Oct 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

kunal-vaishnavi commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Validation

Motivation + Context

Uh oh!

Tianlei Wu (tianleiwu) left a comment

Choose a reason for hiding this comment

Review summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Baiju Meswani (baijumeswani) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kunal-vaishnavi commented Oct 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kunal-vaishnavi commented Oct 12, 2026 •

edited

Loading