INT3 QMoE CUDA correctness baseline and Qwen experimental results by jiafatom · Pull Request #33519 · microsoft/onnxruntime

David Fan (jiafatom) · 2026-10-21T18:45:24Z

Summary

Stacked on #33394 (base: jiafa/int3-qmoe-p0-format-fixtures).

Implement actual CUDA QMoE v1 execution for (FC1, FC2)=(3,3),(3,4),(4,3), FP16/BF16, block32/64/128.
Add row-local cross-byte packed extraction, fused gate/up SwiGLU, independent down projection, routing, optional biases/zero points/aggregation weights, strict content checks, and bounded activation tiling.
Keep effective-INT3 execution separate from legacy INT2/INT4/INT8 paths; retain raw initializers and prohibit all-expert floating-weight materialization.
Add independent CUDA tests, formal OpTester cases, actual CUDA Session fixture/harness code, and a detailed experimental results report with machine-readable measurements.

Not PR2-PR4 Completion

This is deliberately Draft. It delivers a correctness baseline and partial packed-decode work, not all planned stages:

Plan scope	Current status
PR2 correctness	Substantial synthetic/actual-Session evidence; clean target build and formal OpTester execution still required
PR3 packed decode	Direct packed FC1/FC2 extraction implemented; versioned private prepack, cache identity/lifetime and prepack-cost qualification missing
PR4a baseline prefill	Functional bounded direct-packed prefill; required selected-expert/tiled-dequant plus existing GEMM not implemented
PR4b/4c optimized prefill	Not delivered; current INT3 prefill is substantially slower than INT4

Full-model measurements cover all-expert (3,3) against (4,4). Mixed pairs have small-fixture coverage, not full-model qualification. Companion producer PRs are Olive #2749 and Mobius #783; those changes are not included in this ORT diff.

Experimental Model Results

Qwen3-30B-A3B revision ad44e777bcd18fa416d9da3bd8f70d33ebb85d39, block64, FP16, attention/embedding INT4, head INT8. Audited all 48 QMoE layers and external-data bindings.

Model	Graph + Weights	MMLU pipeline check (57 questions only)
All-expert INT3	12,782,047,512 bytes	40/57 (70.18%)
All-expert INT4	16,405,926,354 bytes	49/57 (85.96%)

Model-size reduction: 22.09%. MMLU check is 5-shot, seed1234, first one question per subject, batch1, max4096, no chat template. All 57 document/prompt/target hashes match. These are not full MMLU scores. Full 57-subject evaluation has no final report yet.

A100-SXM4-80GB, batch1, output32, warmup1, repeats3, same runtime and 64MiB scratch:

Input	Bits	TTFT p50 ms	Prefill token/s	Decode token/s	Sampled Process Peak GiB
128	3	901.41	142.18	45.56	13.44
128	4	30.59	4399.88	114.39	31.78
512	3	3544.41	144.51	44.62	14.46
512	4	61.58	8541.29	108.80	31.80
2048	3	14099.40	145.27	42.75	14.45
2048	4	173.88	11914.46	100.36	31.80
4096	3	28184.65	145.34	40.69	16.46
4096	4	335.17	12287.25	90.14	34.81

INT3 saves capacity but is slower. No performance acceptance/speedup claim. NVML process residency is not exact allocator peak; INT3/128 had a 53.84ms sampling gap. Existing INT4 prepack/workspace behavior remains unchanged.

Validation and Build Caveats

Packed loader: 80 CUDA cases; double projection: 216 cases, repeated three times; memcheck zero errors.
82 actual CUDA-only Session cases, including malformed content rejection; rerun before submission. Representative actual Sessions passed memcheck.
Six Olive -> Mobius -> ONNX -> CUDA cases passed for both dtypes/three width pairs, repeated three times, with independent references.
Python reference: 223 passed / one opt-in skip; companion Olive tests: 272 passed; Mobius tests: 170 passed.
Full models completed prefill/decode, eight matched performance runs, and paired diagnostic evaluation.

Not a clean build: isolated CUDA12.8 experiments reused cached core/bindings and runner objects. External-loader and CutlassGemmConfig ABI mismatches were diagnosed and aligned in the experimental build. This is not a production ABI solution. Formal OpTester cases compile but were not reliably executed due cached helper ABI drift. Standalone CUDA tests are not integrated in the normal test target. Full-model sessions allow CPU shape ops.

See docs/design/int3-qmoe-cuda-baseline-results.md and the adjacent JSON for full protocols, provider hash, raw benchmark trials and uncovered scope. No model weights or dataset question text are included.

Remaining Gates

Clean full CUDA build and formal provider/legacy regression tests
Versioned private prepack and immutable initializer validation caching
Required GEMM-based bounded prefill and optimized decode
Capture/concurrency/lifetime qualification
Full matched MMLU and all-combination model qualification

Co-authored-by: jiafatom <30608893+jiafatom@users.noreply.github.com>

Signed-off-by: David Fan <jiafa@microsoft.com>

) ## Summary Implements the P0 reference/tooling and P1 QMoE v1 contract/host-validation slices of the [INT3 QMoE delivery plan](https://github.com/microsoft/onnxruntime/blob/2b1185e24a5a79c56fd01226ad81b016450d0ef3/docs/design/int3-qmoe-end-to-end-delivery-plan.md). Related roadmap: microsoft#32657. No P2 CUDA kernels are included. ### P0: Portable Format and Numerical References - Generic row-local LSB-first UINT8 pack/unpack for widths 1..8, checked size arithmetic, and separate canonical-padding validation. INT5/INT6 utility coverage does not enable those widths in QMoE. - INT3 offset-4 decoding, deterministic fixture RTN with stored-scale rounding and subnormal-underflow reporting, and handwritten byte/numerical vectors covering row/expert boundaries, packed block zero points, and a tiny predetermined-route SwiGLU MoE. - Independent scalar oracles and standard opset-18 ONNX unpack/decode graphs; no draft Pack/Unpack operator is required. - Self-contained reference tests use local golden vectors and installed standard ONNX operators; no external ONNX source checkout or pinned ONNX commit is required. - Reference pytest coverage runs in the existing Python quantization CI stage; CMake copies the helper modules and golden JSON fixture into the build output. No feature-specific workflow is added. ### P1: Compatible QMoE v1 Extension - INT3 is added to QMoE version 1. There is no QMoE v2 registration and no Microsoft-domain available/released-version increase. - Models without an effective INT3 projection keep legacy shape/type propagation. Existing 2/4/8-bit byte semantics and attribute defaults are preserved, including expert_weight_bits=4, quant_type="int", and weights_prepacked=-1. - INT3 requires integer quantization, explicit weights_prepacked=0, activation_type="swiglu", fused interleaved gate/up, block sizes 32/64/128, block-aligned H/I, and matching FP16/BF16 activation/scale types. Omitting quant_type uses the existing "int" default; exporters should still emit it explicitly. - Support independent projection metadata, including (3,4), (3,3), and (4,3), and packed per-row block zero points. Infer H from activation shape and I from fused FC1 output rows, not FC2 packed bytes. Check shapes, types, symbolic dimensions, byte counts, and overflow. - Shared host-only ValidateInt3ProjectionContent requires block-aligned K (not N) at the projection level and checks exact buffer sizes, finite nonnegative FP16/BF16 scales, all-4 explicit zero points, canonical padding, and checked dimensions. Padded byte-count arithmetic is overflow-safe without changing the strict byte-aligned helper. - Native regressions verify that legacy INT4 models resolve to QMoE v1 and find the existing CPU kernel both with and without an explicit Microsoft-domain import. Adjacent Microsoft operators remain at v1. - Generated QMoE documentation is synchronized; non-QMoE documentation is unchanged. ## Validation of the v1 Revision Commit eda7353 was validated inside Docker jiafa-dev: - Offline Python reference tests: **223 passed, 1 skipped**. The historical optional external-checkout case was removed in 6c875ae; current results are listed below. ONNX 1.23.2, NumPy 2.4.6, ml_dtypes 0.6.0; BF16 coverage executed. - Actual current ORT schema, registration entry point and QMoE test sources compiled using cached Ninja-generated flags and linked with cached ORT libraries: **17 passed** (11 schema/registry tests and 6 content/byte-helper tests). - C++ filter: QMoEShapeInferenceTest.*:QMoEFormatValidationTest.*. New test files are collected by the existing contrib-test CMake glob. - Pinned clang-format 20.1.8, Ruff 0.12.12 check/format, editor diagnostics and git whitespace checks passed. - QMoE description was checked byte-for-byte using the documentation generator's rendering rules; all other operator documentation is byte-identical to the preceding PR head. - This is isolated current-source compilation/linking, **not a clean full ORT build** or CUDA inference qualification. The cached build predates two neighboring schemas, so their current definitions were also compiled for linking. Container CUDA runtime paths and system -lrt replaced unavailable cached paths. Only the existing MLAS GNU-stack linker warning remained. - Hosted CI for this new head is pending. Previous external-reference interoperability results are historical; that opt-in test was not rerun for this schema-only revision. ## Scope and Review Gates This PR adds **no INT3 execution kernel**. Existing v1 CPU/CUDA/WebGPU registrations remain; schema acceptance does not establish INT3 inference support. Provider validation/execution, packed CUDA decode, bounded prefill, export and model qualification belong to subsequent work. The end-to-end follow-up now builds on this v1 revision; see the coordinated Draft PRs below. The host content validator has **no runtime call sites yet**: it accepts caller-supplied host buffers and is directly unit-tested. Schema inference checks metadata, not weight contents; model loading does not automatically scan scales or zero points. Python-only tails/empty dimensions/FP32-scale fixtures do not broaden the block-aligned FP16/BF16 operator profile. The v1-compatible extension approach supersedes this PR's earlier v2/domain-version proposal. Operator/exporter owner review and format freeze remain required. No approved ONNX datatype, standard operator, production quantizer/exporter, checkpoint importer, accuracy or performance claim is made. ## Scoped Checkpoint Audit The local Olive audit at a9290ab9bf668cae5a02ad7473c8cb0f5476d35e found WeightQuantizer and pack_to_uint8 restricted to 2/4/8 bits, with packing_factor = 8 // bits; the inspected GPTQ pass imports that quantizer, and PrecisionBits has no three-bit option. These inspected paths do not establish a compatible native INT3 checkpoint source. External GPTQ implementations and GGUF Q3_K/IQ3 were not audited. Future importers must verify code mapping, bit order, tensor orientation, grouping, scales, zero points, permutations, gate/up interleaving and FC2 orientation rather than relabel unknown checkpoint bytes. Unrelated local files and submodule changes are excluded. ## Follow-up End-to-End Work The experimental end-to-end implementation now builds on this PR's compatible QMoE v1 contract. The work is split across three coordinated **Draft PRs**: - **ORT runtime: [microsoft#33519](microsoft#33519, stacked directly on this PR. Adds CUDA execution for `(FC1, FC2)=(3,3), (3,4), (4,3)` with FP16/BF16, direct packed extraction in both projections, routing and fused SwiGLU, strict validation, bounded activation tiling, tests, and experimental model results. - **Olive quantization: [microsoft/Olive#2749](microsoft/Olive#2749. Adds symmetric grouped INT3 RTN with offset-4 codes, stored-scale rounding, canonical row-local three-bit packing/unpacking, and ceil-bit storage shapes. - **Mobius export: [onnxruntime/mobius#783](onnxruntime/mobius#783. Enables expert-only INT3 overrides for the three width pairs while retaining the model-wide INT4 fallback, using existing native fused QMoE graph and external-data plumbing. Together these changes exercised source checkpoint -> Olive quantization -> Mobius ONNX export -> ORT CUDA inference. Six small cross-project cases cover both activation dtypes and all three width pairs. Full Qwen3-30B-A3B exports and model measurements cover all routed expert gate/up/down weights at INT3 `(3,3)` versus matched INT4 `(4,4)`, with 48 QMoE layers, block64, FP16, attention/embedding INT4 and head INT8. Graph-plus-weight size is 12,782,047,512 versus 16,405,926,354 bytes, a **22.09% reduction**. The runtime PR records A100 measurements at input lengths 128/512/2048/4096 and output length 32. **Current INT3 inference is substantially slower than INT4; this is not a performance delivery.** The paired 5-shot MMLU pipeline check uses only one question per subject (57 total): INT3 40/57 and INT4 49/57, with matching document/prompt/target hashes. These are **not full MMLU scores**; no completed full-MMLU result is claimed here. ### Delivery Plan Status These follow-ups do **not** complete planned PR2 through PR4: | Planned stage | Implemented evidence | Remaining gate | | --- | --- | --- | | PR2: CUDA correctness | Independent references, bounded execution, actual CUDA Session coverage | Clean provider-target build and formal integration-test acceptance | | PR3: packed decode | Direct packed FC1/FC2 execution for all three pairs | Versioned provider-private prepack, cache identity/lifetime and prepack-cost qualification | | PR4a: baseline prefill | Functional bounded direct-packed prefill, including full-model long prompts | Required selected-expert/tiled dequantization plus existing GEMM and dedicated acceptance tests | | PR4b/4c: optimized prefill | Not delivered | Winning kernel candidate, integration and performance acceptance | Before submission, the follow-up runtime's 82 actual CUDA Session cases passed; Olive's 176 scoped quantization tests and Mobius's 18 new tests also passed. Runtime experiments reused isolated cached build artifacts with explicit ABI alignment, **not a clean full build**. Formal OpTester execution, capture, concurrency, lifetime and full-model qualification remain open. Full-model mixed-pair qualification is also open; small fixtures do not close that gate. See microsoft#33519 and its committed results report/JSON for protocols, provenance, measurements and limitations. The checkpoint audit above describes the inspected pre-change Olive revision; microsoft#2749 supplies the subsequent producer changes. None of these follow-ups changes this PR's P0/P1-only diff or constitutes ONNX standard approval or production qualification. ## Review Fixes and Padding Experiment Follow-up Commit e45296b integrates the no-exceptions assertion guards, symbolic projection-row consistency fix and regression cases, and unpacked zero-point code clarification directly into this PR. Rebuilt current-source normal schema/format tests: **17/17 passed**; format tests compiled with ORT_NO_EXCEPTIONS plus the real helper implementation: **6/6 passed**. Pinned clang-format, whitespace and schema/doc synchronization checks passed. These remain isolated cached-dependency builds, not clean full or globally exception-disabled builds. Padding experiments remain separately reviewable in microsoft#33530: two corrected full A100/CUDA13 runs, 108 cases each, with all128expert rotation covered per full trial. For block64,24->32byte padding adds33.33% weight payload and is approximately11% slower for decode projection and10% slower for32token projection in the tested scalar-load implementation. This is not a whole-model or pure-bandwidth conclusion and does not rule out future vectorized private layouts. The padding design thread remains open for reviewer acceptance; experiment source and raw records are not added to this P0/P1 diff. ## Existing CI Integration Commit 8a41267 removes the dedicated packed-reference workflow and runs the suite from the existing Python quantization test stage in tools/ci_build/build.py, after unittest discovery when contrib ops are enabled. CMake copies the complete reference directory and golden_vectors.json into the test build output. Explicit pytest execution is necessary because unittest discovery does not collect these module-level pytest functions. Existing Python CI already installs pytest; no new workflow or dependency-install job is introduced. Validated by copying the directory with cmake -E copy_directory and executing the exact new pytest command extracted from build.py: **223 passed, 1 skipped** at 8a41267 (historical optional checkout test, removed in 6c875ae). Ruff check/format and git whitespace checks passed. Hosted CI for this head is not yet verified. ## Latest Review Fixes Commit 6c875ae addresses the latest review comments: - Removes the pinned external ONNX reference checkout test entirely. The suite is self-contained: **223 passed**, with no external-checkout skip. The ONNX submodule gitlink is unchanged by this PR. - Requires explicit `activation_type="swiglu"` for the INT3 fused profile and rejects the default/non-SwiGLU cases. Legacy INT4 activation behavior remains covered. Schema documentation is synchronized. - Removes the projection-level N-alignment restriction (K remains block-aligned), removes unreachable weight-tail validation, and computes each expected zero-point byte once before checking rows. Adds an N=127 acceptance regression. - Adds standard copyright/license headers and trailing newlines, restores the unrelated CPU model test to the main-branch version, and fixes CMake COMMAND indentation. Recompiled current affected C++ sources with cached dependency libraries: **18/18 normal schema/format tests passed** and **7/7 format tests passed with ORT_NO_EXCEPTIONS and the real helper implementation**. These are scoped cached-dependency builds, not a clean whole-ORT or globally no-exceptions build. The exact existing-CI pytest command also passed after CMake-copying the suite into a temporary build-output directory. Pinned clang-format, Ruff check/format, schema/doc synchronization and git whitespace checks passed. Hosted CI for this new head has not yet been verified. --------- Signed-off-by: David Fan <jiafa@microsoft.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>

## Summary Narrowly scoped producer-validation hardening and regression coverage on top of the existing native INT3 support in #2716. This PR does not introduce a new packing format or quantization algorithm, and is not a prerequisite for the ORT runtime changes in microsoft/onnxruntime#33394 or microsoft/onnxruntime#33519. - Preserve the existing unsigned symmetric INT3 RTN scale policy: `max(abs(weight)) / 3.5`, implicit zero point 4, and the existing all-zero-group scale `1 / 3.5` with code 4. Group sizes 32/64/128 do not implicitly select a different producer policy. - In `WeightQuantizer.find_qparams`, reject non-finite weights or computed scales, and a scale that rounds to exactly zero for a nonzero unsigned symmetric INT3 group, regardless of group size. Nonzero subnormal stored scales remain allowed with unchanged dtype-rounding limitations. - Limit this fail-fast producer policy intentionally to unsigned symmetric INT3: report an unrepresentable automatically computed scale rather than silently clamping it to the minimum positive value, without broadening this PR to error-handling changes for other widths or modes. This policy is not an ORT requirement. - Describe the validation boundary explicitly: consumers of `find_qparams`, including `fake_quantize`, inherit these checks. `QuantTensor.from_float` bypasses them when both scales and zero points are supplied. This is not universal INT3 or checkpoint validation. - Exercise existing row-local LSB-first three-bit packing, cross-byte codes, tail padding, checkpoint placeholders, and independent quantization-to-golden-bytes regressions. Retain the existing shared packed-shape guard; remove the redundant INT3-specific duplicate. - Document the retained producer policy and ORT mapping `(code - 4) * scale`, subject to runtime-supported layouts, shapes, dtypes, and valid scales. Remove the withdrawn formula from permanent documentation. ## Validation 788 tests passed with the normal root conftest: ```bash pytest test/common/quant/test_int3.py test/common/quant/test_utils.py test/common/quant/test_quantizer_numerics.py -q ``` This includes the existing incompatible-shape/shared-error regressions and four new dtype-specific nonzero-subnormal-scale regressions. Touched-file Ruff lint and formatting checks, plus `git diff --check`, passed. This is targeted verification, not the full suite. No model-quality or performance improvement is claimed. Earlier experimental exports used a previous producer policy and do not qualify the revised code. This PR does not enable native INT3 ONNX export or imply general dense MatMulNBits INT3 support, checkpoint-wide validation, or end-to-end ORT/runtime qualification. --------- Signed-off-by: David Fan <jiafa@microsoft.com> Co-authored-by: jiafatom <jiafatom@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

David Fan (jiafatom) and others added 10 commits October 19, 2026 17:25

Add P0 packed-code references and INT3 QMoE golden fixtures

caaecfe

Add versioned INT3 QMoE contract and host validation

f0d5d76

Fix QMoE review findings and portable CI regressions

2bacbf0

Update generated QMoE version 2 documentation

03d8868

Add trailing newline to QMoE reference workflow

d3e6f4a

Align QMoE schema documentation indentation

a9fb837

Co-authored-by: jiafatom <30608893+jiafatom@users.noreply.github.com>

Sync generated operator documentation

afdf182

Co-authored-by: jiafatom <30608893+jiafatom@users.noreply.github.com>

Fix generated QMoE documentation indentation

1e034f2

Co-authored-by: jiafatom <30608893+jiafatom@users.noreply.github.com>

Extend QMoE v1 with the INT3 profile without a domain version bump

eda7353

Add INT3 QMoE CUDA correctness baseline and experimental results

fbdc430

Signed-off-by: David Fan <jiafa@microsoft.com>

This was referenced Oct 21, 2026

Harden native INT3 validation and regression coverage microsoft/Olive#2749

Merged

Allow canonical Olive INT3 expert overrides in native QMoE export onnxruntime/mobius#783

Draft

P0/P1: packed-code foundation and INT3 QMoE v1 contract #33394

Merged

Base automatically changed from jiafa/int3-qmoe-p0-format-fixtures to main October 22, 2026 22:41

This was referenced Oct 22, 2026

2, 3, 5, 6 bit quantization exploration #32657

Draft

Add INT3 QMoE CUDA execution and versioned private prepack #33555

Closed

Navigation Menu

Repository navigation

INT3 QMoE CUDA correctness baseline and Qwen experimental results - #33519

INT3 QMoE CUDA correctness baseline and Qwen experimental results#33519
David Fan (jiafatom) wants to merge 10 commits into
mainfrom
jiafa/int3-qmoe-end-to-end

David Fan (jiafatom) commented Oct 21, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

David Fan (jiafatom) commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Not PR2-PR4 Completion

Experimental Model Results

Validation and Build Caveats

Remaining Gates

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

David Fan (jiafatom) commented Oct 21, 2026 •

edited

Loading