[Akshay Sonawane (apsonawane)]

Summary

Add standalone package assembly and generic contrib-operator rewrites, with optimizer/builder tests and upstream-compatible numeric CUDA provider options.

Supersedes #2756. Reconstructed directly on current main (bf7aaea80e4f902ebdc38c49fa86e1617ea5f41f), without inherited feature-branch history. This PR independently targets main; there is no Git/build stack dependency between the two GenAI replacements. Executing generated model packages requires the graph runtime and compatible companion ORT operators microsoft/onnxruntime#33469 through microsoft/onnxruntime#33474.

Unrelated inherited builders/model features, replay/allocator changes and API-wrapper changes are deliberately excluded. No benchmark/profiling/evaluation drivers or local task metadata are included.

Package assembly review fixes

• Reject existing active bundle destinations before copying anything, including later-destination collisions and dangling symlinks. Rebuild into a fresh output directory rather than mixing old graph/weight files with new configuration and libraries.
• Preserve directory symlinks without traversing them, alongside existing file-symlink support.
• Inventory shared weights from the original source graphs, using device/inode/range/type/shape identity. Destination pool paths still point at packaged files, preserving sharing through --copy and cross-filesystem hard-link fallback. Same-name unrelated source weights remain distinct.
• Extract QSA normalization and rotary constants exclusively from the packaged target/head graphs, without an implicit research-machine reference override.
• Remove the three CodeQL-reported sys.exit(main()) uses of always-None procedure returns. Check the v12b provider before assembly, retaining dependency-free CLI help.
• Document safe rebuild behavior, source sharing, symlink reachability and CPU test commands.

Validation and remaining qualification

• Used the qwen-flash Conda environment.
• Nine CPU package-builder regressions pass, including forced EXDEV fallback and source-identity negative controls; all three builder CLI help checks pass.
• Optimizer regressions pass: 29 tests and 85 subtests. Added capture/unnamed-node, contrib-import, rotary collision, unsafe PostMix, optional-output and transformed-sharing cases.
• Optimizer fixes protect subgraph-captured values/initializers across rewrites, use node identity instead of optional names, preserve transformed shared weights through source-pool/transformation provenance, reject invalid/unsafe PostMix fusion, add missing contrib opset imports, and allocate collision-free rotary names.
• Full-PR lintrunner -a and check-only lint pass in qwen-flash; whitespace checks pass.
• Per request, no ORT/GenAI rebuild or native/GPU/model execution is performed for these review fixes.
• Draft: full native compatibility, compatible package load and end-to-end Generator/model parity remain outstanding. Earlier combined-branch results do not qualify these independent branches. Synthetic graph and builder tests do not establish package/model accuracy or performance. PLE construction still requires the external research dependency and compatible checkpoints.

[Copilot]

🟡 Changes recommended

Unresolved graph-validity, package-consistency, and weight-sharing defects can prevent correct loading or execution.

11 open issues

• Reject existing bundle destinations before copying · New
• Preserve directory symlinks in bundled weights · New
• Preserve weight sharing identities when copying bundles · New
• Use packaged target constants instead of hard-coded model · New
• Protect captured subgraph values before pruning · New
• Use unique identifiers instead of optional node names · New
• Preserve sharing for transformed weights · New
• Skip LowRankMix nodes with existing streams output · New
• Preserve topological order when moving mixed producers · New
• Add contrib opset imports for introduced nodes · New
• Generate collision-free prefixes for rotary fusion · New

What changed in this PR

Adds standalone Qwen graph packaging and optimization tools to ONNX Runtime GenAI. Generated packages require the companion graph runtime and compatible ORT operators.

Changes:

• Adds package builders with numeric CUDA provider options.
• Adds graph rewrites and weight-pool metadata handling.
• Adds synthetic optimizer tests and packaging documentation; full runtime qualification remains pending.

File	Description
tools/​qwen38_graph/​test_optimize_ported_package.py	Tests rewrite behavior, guards, and source-file preservation.
tools/​qwen38_graph/​PACKAGING.md	Documents tooling, dependencies, and validation limits.
tools/​qwen38_graph/​optimize_ported_package.py	Implements graph optimization and package metadata updates.
tools/​qwen38_graph/​build_package.py	Assembles separate target/head packages.
tools/​qwen38_graph/​build_package_v12b.py	Adds library and CUDA provider selection.
tools/​qwen38_graph/​build_package_v12.py	Assembles folded packages with prefill and shared weights.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Unresolved assembly and graph-rewrite defects can abort construction or produce invalid packages.

13 open issues

• Reject output destinations inside source bundles · New
• Read QSA constants to EOF when length is absent · New
• Handle inline BF16 tensors stored in typed fields · New
• Handle external data without a length field · New
• Skip fusion for non-FP32 weight scales · New
• Prevent generated initializer name collisions · New
• Recognize fused FP8 projections during output quantization · New
• Rename captured control-flow values recursively · New
• Skip empty pool entries before indexing fields · New
• Validate normalization scale layout and dtype before fusion · New
• Default omitted normalization epsilon to 1e-5 · New
• Ensure fused inputs precede the replacement node · New
• Prevent early consumers of retained history outputs · New

11 resolved since last review

• Generate collision-free prefixes for rotary fusion
• Add contrib opset imports for introduced nodes
• Preserve topological order when moving mixed producers
• Skip LowRankMix nodes with existing streams output
• Preserve sharing for transformed weights
• Use unique identifiers instead of optional node names
• Protect captured subgraph values before pruning
• Use packaged target constants instead of hard-coded model
• Preserve weight sharing identities when copying bundles
• Preserve directory symlinks in bundled weights
• Reject existing bundle destinations before copying

_{Review effort: Balanced}

[Akshay Sonawane (apsonawane)]

Pushed 438cb21 and addressed all 13 new review threads.

• Reject resolved bundle destinations nested inside any source before writing.
• Support optional external-data lengths and typed inline BF16 QSA constants.
• Require FP32 FP8 scales, allocate collision-free generated names, and preserve transformation provenance.
• Support optimized FP8 head producers with missing shapes; reject captured head outputs explicitly before quantization.
• Round-trip empty shared-weight pools.
• Guard hyper-connection scale type/layout and input availability; apply the source epsilon default.
• Preserve convolution carry-output consumer ordering, including nested captures.

Validation: all 46 CPU-only Python packaging/optimizer tests pass; full-PR lintrunner apply/check-only and git diff --check pass. Updated packaging documentation. No local builds, native compilation, GPU execution, or full-model qualification were performed.

No checks were failed on the previous head at inspection (37 successful, 17 pending/running). Fresh CI is triggered by this push and remains unverified, not claimed green.

[Copilot]

🔵 Needs a closer look

Library filename collisions and relative-symlink handling can produce packages that cannot load required operators.

No open issues 🎉

3 previously missed issues

In code that hasn't changed since last review

Resolve library symlinks before creating hard links

tools/​qwen38_graph/​build_package.py:81

On Linux, a relative library symlink such as libcustom.so -> libcustom.so.1 can take this successful hard-link path and copy the symlink itself. Since its target is not packaged alongside it, the destination is dangling and the operator/provider library cannot load. Resolve the source before hard-linking, matching the behavior of --copy. Bundle symlinks remain preserved by the separate handling in link_tree.

Reject conflicting library basenames during bundle copying

tools/​qwen38_graph/​build_package.py:160

Distinct --lib-* inputs such as /qsa/libcustom_ops.so and /hc/libcustom_ops.so both become libs/libcustom_ops.so. The later copy replaces the first, while both registration entries reference the remaining library, losing required operators. Reject conflicting destination filenames before copying bundles, including the reserved libqwen38_plane.so filename. Allow repeated references to the same source file.

Reject conflicting library basenames in folded builds

tools/​qwen38_graph/​build_package_v12.py:239

The folded builder also maps distinct libraries with the same basename to one destination. For example, /qsa/libcustom_ops.so and /hc/libcustom_ops.so overwrite each other, leaving both registration entries pointing to the last library and losing required operators. This also affects the v12b wrapper. Reject conflicting filenames before copying bundles, including libqwen38_plane.so, while allowing repeated references to the same source file.

13 resolved since last review

• Prevent early consumers of retained history outputs
• Ensure fused inputs precede the replacement node
• Default omitted normalization epsilon to 1e-5
• Validate normalization scale layout and dtype before fusion
• Skip empty pool entries before indexing fields
• Rename captured control-flow values recursively
• Recognize fused FP8 projections during output quantization
• Prevent generated initializer name collisions
• Skip fusion for non-FP32 weight scales
• Handle external data without a length field
• Handle inline BF16 tensors stored in typed fields
• Read QSA constants to EOF when length is absent
• Reject output destinations inside source bundles

_{Review effort: Balanced}

[Akshay Sonawane (apsonawane)]

Added a focused packaging-documentation update for the experimental native decode preset.

Published commit: 60d20ed7c150944391d5fa9adbd7711a6c96c375.

• Documented GDN/convolution, per-column FP8, compact QSA, vectorized INT8 attention, shared-factor attention merge, and warp-selection routing options.
• Linked companion ORT PRs [CUDA] Extend GatedDeltaNet and CausalConvWithState for S<=8 speculative decode onnxruntime#33471, [CUDA] Extend DynamicSparseAttention INT8 KV cache and PackedSparseAttentionIndexer QSA onnxruntime#33472, and Optimize quantized MoE and block-scaled FP8 CUDA projections onnxruntime#33473, with exact-dispatch documentation remaining in ORT.
• Clarified that environment flags require a compatible native build and do not add GenAI runtime features.
• Documented preserving qualified packages and avoiding in-place writes through hard links.
• Retained qualification limits: no full accuracy claim, no sustained 250 TPS at 8K, and no clean-full-suite claim. Focused publishing-version native tests pass (175 off + 175 on); a broader GDN empty-request failure is reproducible with the unmodified ORT PR-head kernel in the incremental retained-dependency build.

This update intentionally contains no GenAI runtime/engine changes, model binaries, weights, machine-specific benchmark drivers, or accumulated unrelated experiments. No new GenAI PR is needed for these native optimizations; the existing graph runtime PR #2758 remains unchanged.

Validation: documentation-only diff and whitespace check. Native incremental compilation/testing and benchmark provenance are detailed on the companion ORT PRs; clean-build CI and full accuracy qualification remain outstanding.

[Akshay Sonawane (apsonawane)]

Published the scoped convolution carry-state fusion in f44e754.

• Reuses the existing CausalConvWithState final/first-state outputs; no new ORT operator.
• Absorbs private carry transposes and first-token Slice/Concat chains, with rank/layout/history/dilation/default-attribute and graph-capture guards.
• Preserves all existing PR-head fixes; no wholesale replacement from the experimental branch.
• Updates the optimizer pipeline, rewrite metadata, tests, and portable packaging documentation.
• Exact updated PR snapshot: 47 packaging/optimizer tests passed; Ruff checks passed.
• Paired experimental captures measured six fewer graph nodes and four fewer GPU kernels per verification cycle (1046 -> 1042), with 266 cycles per capture.
• Twenty-four measured requests across 2K/8K/16K matched all 512 output tokens against the unfused baseline, with no generation failures.

Additional integration evidence from a fresh build of the complete experimental ORT/GenAI sources:

Prompt	Pooled decode TPS	Mean TTFT
2048	269.90	0.584 s
8192	250.17	2.448 s
16384	266.92	5.065 s
32768	264.12	10.578 s

One warmup plus three measured requests per context, CUDA Graph enabled, 512 outputs/request, greedy decoding on H200. All 16 requests matched the previous-runtime token IDs; zero standard fallbacks, MTP failures, or DFlash2 failures. Loaded-library inspection verified the freshly built runtime/provider/bindings.

Scope/limitations: those timings use the complete experimental source stack, not this tooling PR in isolation or a fresh paired Tianlei comparison. They do not establish repeatability across fresh processes. Full accuracy qualification remains deferred. The complete native stack has 333 passing ORT tests, 69 passing selected GenAI tests (including real-GPU mapped weights), and two previously known malformed-state QSA/CSA failures; the latter also fail with specialized top-k disabled. One GenAI test remains disabled. This is not a blanket full-suite or deployment qualification.

[Akshay Sonawane (apsonawane)]

Pushed b709837: guarded shared-expert projection and RMS-normalization/rotary rewrites, focused shape/consumer/hardlink/atomic-replacement tests, and packaging documentation. Adapted to the reviewer-fixed optimizer/type inference rather than replacing files wholesale. All 36 staged optimizer tests pass. Requires matching companion ORT kernels (#33469, #33470, #33473); no new third-party operators are emitted. Full accuracy is not yet qualified: beginning uncapped greedy LiveCodeBench, first 10 smoke problems then the full pinned release_v6 suite.