[Akshay Sonawane (apsonawane)]

Summary

Optimize CUDA sparse paged attention, packed sparse indexing, and quantized MoE execution for Qwen workloads.

This PR now targets main. Main was merged with a normal merge commit; the existing PR history is preserved without force-pushing. The final diff contains the original optimization scope and necessary integration fixes, not unrelated inherited feature-branch model-package, WebGPU, platform, allocator, or model changes.

Changes

Sparse paged attention

• Add tiled grouped-query attention with increased key/value reuse.
• Add opt-in tensor-core QK and PV paths for eligible prefill shapes.
• Preserve the older opt-in grouped-prefill path and the general fallback.
• Retain upstream attribute validation, suppressed-write prefix-cache reads, INT8 current-token cache semantics, and flattened launch dimensions.
• Extend numerical/reference coverage and add a tiled suppressed-write regression.

Packed sparse attention indexer

• Improve hierarchical scoring and top-K selection.
• Reduce work for inactive fixed-capacity state slots.
• Add default-off warp-local scoring and radix-selection paths.
• Retain upstream state/IndexShare validation, dynamic shared-memory checks, and launch bounds.
• Add tile-boundary, stable-tie, and packed-row regression coverage.

Quantized MoE

• Optimize FP8 row-scale access and weight decoding while preserving upstream native FP8 decoding and FP32 outputs.
• Add opt-in packed NVFP4 prefill with fused SwiGLU, routing-scratch reuse, and tiled down-projection output.
• Avoid grouped-GEMM tactic profiling on standalone FP4 GEMV and packed-prefill paths that do not consume it.
• Include default-off INT2 experimental paths and extended CUDA/Python regression coverage.

Main integration validation

• Used the qwen-flash Conda environment.
• Ran lintrunner -a before the merge commit, followed by a passing check-only run on the complete PR delta.
• Compiled affected sparse-paged-attention and packed-indexer CUDA/host sources and C++ test translation units against the main-based source tree.
• Compiled affected QMoE/packed-prefill CUDA and internal-test translation units; host QMoE also compiled in plugin and non-plugin configurations.
• CUDA compilation targeted SM90; Python syntax and whitespace checks passed.
• No benchmark/profiling/evaluation scripts or local task metadata were added.

These are historical compile/static checks from the main merge, not validation of the later review fixes or a fresh executed GPU/model qualification.

Review fixes and current validation

• Reuse the common main-candidate resolver in grouped prefill so negative slot mappings suppress writes without dropping mapped prefix-cache reads.
• Correct the mixed-source attention reference and make cached versus current values deliberately different for the suppressed-write row.
• Add isolated-process reference comparisons for FP16 scalar, QK-only, QK+PV, BF16 QK, and grouped scalar/vectorized loads. Cases cover multiple/partial tiles, empty/invalid selections, mapped prefix hits, and attention sinks. Assert verbose dispatch messages to reject unintended fallback execution.
• Add isolated-process radix-selection flag-off/on reference coverage; see the indexer regression documentation.
• Used the qwen-flash Conda environment, ran lintrunner -a before committing, followed by check-only lint and Python syntax/whitespace checks.
• No ORT/GenAI builds or native/GPU test execution were performed for these fixes, as requested. Added regression tests require execution against a build containing the new head.

Pipeline triage

The previous head had no failed GitHub Actions build jobs to repair: only Copilot and CLA succeeded, and the Azure QNN check was action_required with the stale message "Skipped due to merge conflicts." GitHub now reports the PR mergeable against current main. The review-fix push retriggers normal CI; green pipeline status remains to be confirmed on the new head.

Earlier feature-branch validation (historical)

The earlier feature-branch implementation was built with CUDA 13.3/FP4 QMoE on H200:

Coverage	Earlier result
Sparse attention, packed indexer, and QMoE operator tests	147 passed, 24 skipped, 2 failures
Internal FP8/NVFP4 CUDA tests	5 passed
Packed NVFP4 prefill Python tests	137 passed
lintrunner	Passed

The two malformed-state indexer failures were observed against both the earlier candidate and its unchanged feature-branch target provider:

• PackedSparseAttentionIndexerTest.QsaMalformedStateLengthsAreClamped
• PackedSparseAttentionIndexerTest.CsaMalformedStateLengthsAreClamped

This is not evidence of their status against current upstream main, and those historical test counts do not qualify this new main-integrated tree.

Scope and limitations

• Experimental paths remain opt-in or default-off.
• No GenAI runtime changes are included.
• No new end-to-end throughput, memory, or model-quality claim is made.
• Some packed-indexer and QMoE optimization hunks overlap the corresponding newer operator PRs [CUDA] Extend DynamicSparseAttention INT8 KV cache and PackedSparseAttentionIndexer QSA #33472 and Optimize quantized MoE and block-scaled FP8 CUDA projections #33473; these PRs are not a required Git stack. Their final landing scopes/order must be coordinated to avoid duplicate changes.

[Copilot]

🟡 Changes recommended

Unresolved prefix-cache correctness issues and missing explicit coverage of experimental kernels block approval.

4 open issues

• Grouped prefill drops prefix-cache hits for negative slot mappings · New
• Reference tests omit tensor-core variants and BF16 QK coverage · New
• Packed sparse radix top-k specialization lacks test coverage · New
• Reference incorrectly excludes cached entries for negative slot mappings · New

1 resolved since last review

• Create calibration capture files with owner-only permissions

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Extensive CUDA dispatch and workspace changes still require current-head GPU qualification and final human review.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Add regression tests for QMoE calibration capture file handling

onnxruntime/​contrib_ops/​cuda/​moe/​moe_quantization.cc:866

No test enables ORT_QMOE_CALIBRATION_CAPTURE_DIR, so the existing QMoE numerical tests do not exercise this new file-writing path. Add capture-enabled regressions using a temporary directory and a named NVFP4 node. Verify the header and tensor payloads, the 16-invocation/128-row limits and exclusion of calls below 16 rows, exclusive-create failure without overwriting an existing file, and rejection during CUDA graph capture. These behaviors affect inference errors and calibration-data validity independently of kernel output accuracy.

4 resolved since last review

• Reference incorrectly excludes cached entries for negative slot mappings
• Packed sparse radix top-k specialization lacks test coverage
• Reference tests omit tensor-core variants and BF16 QK coverage
• Grouped prefill drops prefix-cache hits for negative slot mappings

_{Review effort: Balanced}

[Akshay Sonawane (apsonawane)]

Pushed 8f83777 to address the latest CI failures:

• Removed direct core logging headers from both sparse CUDA implementations. The existing env_var_utils.h include selects core logging for plugin builds while shared-provider builds retain their provider logging facade. This fixes the Linux CUDA/TensorRT macro redefinitions and Windows CUDA logger/Capture type conflicts without changing dispatch or kernel behavior.
• Replaced std::getenv in the radix top-k regression helper with Env::Default().GetEnvironmentVar, fixing MSVC C4996 across Windows CPU, CUDA plugin, ARM64, and WebGPU test builds.
• Guarded the death-test-only helper group with GTEST_HAS_DEATH_TEST, matching its caller and fixing the WASM unused-function error.

All existing inline review threads were already resolved; there were no new unresolved threads at this pass. Full-PR lintrunner -a, check-only lint, and git diff --check passed in the qwen-flash environment. No local builds, compilation, native tests, or model execution were performed, as requested. Fresh CI is pending; these results are not yet claimed green.

[Akshay Sonawane (apsonawane)]

Pushed 3a7dbc1 to address the current pipeline failures:

• Windows CUDA + TensorRT: the linker reported unresolved Env::Default from QMoE calibration capture naming. Replaced the direct core Env call with logging::GetProcessId(), which has an existing shared-provider host bridge, and removed the direct Env header dependency.
• Windows WebGPU: link-time C4744 reported conflicting Abseil FastTypeTag types from the typed GoogleTest death_test_style setter. Switched to the existing repository pattern using Abseil flag reflection, with the non-Abseil fallback retained, and restore the previous style after the test.
• Windows CUDA plugin test: the radix parity child exited with status 1, but its assertion details were lost at std::_Exit. Added named case traces and explicit stderr reporting/flushing of failed GoogleTest parts before exiting. All five cases, both dispatch modes, exact integer comparisons, and dispatch-marker checks remain enabled. The underlying child-test failure is not yet established as fixed; fresh CI will provide the missing diagnostics if it persists.

The WebGPU static summary failure was downstream of the failed compile job; cancelled Linux/vcpkg jobs had no independent failure diagnostics.

Validation: full-PR lintrunner and git diff --check passed. No local native build/test was run, per the PR-fix restriction. Fresh CI results are pending; this is not a claim that the Windows plugin parity failure is resolved.

[Akshay Sonawane (apsonawane)]

Follow-up f851671 fixes the Windows plugin failure now exposed by the diagnostics: the empty_row fixture illegally provided position_ids for QSA, and graph resolution rejected it. Replaced that fixture with a schema-valid two-request batch: past lengths 0 and 8204 yield an empty selection row and a populated row without position_ids. Score-group initialization now covers each request. Both dispatch modes, all five cases, and strict comparisons remain enabled. The previous rerun passed Windows plugin compilation and the linker fixes; lint/diff-check pass for this follow-up. New CI is pending; no local native build was run.

[Akshay Sonawane (apsonawane)]

The latest Windows plugin rerun confirms all parity assertions now pass: the remaining failure was solely the missing dispatch log regex (successful child exit). Fixed that in this follow-up by setting the existing ORT_UNIT_TEST_MAIN_LOG_LEVEL=0 option for each re-executed child. The CUDA plugin caches Ort::Logger severity during startup registration, so changing the core default severity inside the test body was too late. This keeps production logging unchanged and retains both kernel dispatch-marker assertions. Lint/diff-check pass; CI will verify.

[Akshay Sonawane (apsonawane)]

Pushed b7be0b3 to carry the recent automatic NVFP4 packed-prefill behavior in the PR that owns that implementation.

• ORT_QMOE_NVFP4_PACKED_PREFILL now defaults to 1 for eligible configurations; explicit 0 remains the opt-out.
• Existing architecture, row-major layout, aligned-dimension, fused-SwiGLU, routing and fallback gates are unchanged.
• Updated directly related docs and added FP16/BF16 unset-versus-explicit output/dispatch coverage plus opt-out dispatch checks.
• Optional fused-SwiGLU, routing-scratch reuse and tiled-down-output experimental flags remain default-off.

Related INT2 sharing, 512-expert/top-10 eligibility and explicit single-prepacked representation support are pushed separately to #33473 (25d286b), without importing this PR's broader NVFP4 implementation there.

Prior isolated-provider qualification compared the three automatic optimization flags unset versus explicitly enabled across 8K/16K/32512 contexts: all 9,216 measured tokens matched, peak memory matched, and MTP remained healthy. This evidence is from the qualified experiment build, not the exact newer PR head. The new repository regression test has not been run locally.

Per request, no PR-local builds, tests, linters or CI watchers were run. Exact-head CI validation is still required. No model binaries, local package exports, profiling data or unpromoted paired-column kernel experiments are included.

[Tianlei Wu (tianleiwu)]

Superseded by #33472 and #33473 for Primary INT4+INT8 DFlash2. Closing to avoid merge conflicts.