[Tianlei Wu (tianleiwu)]

Stacked on #2820 (base tlwu/20261030/qwen38-package-builders; the diff shows only this change). Retarget to main once #2820 merges (and #2729 below it). Series map: #2726.

The scripts are Python-only and do not touch the runtime. The PR was split out of the old #2729, which carried the package builders and this one together.

Summary

Adds tools/qwen38_graph/build_package_offload_v3.py, which converts a v12b package into a QMoE-offload package for GPUs that cannot hold the 63 GiB of routed experts, and makes the two flat-package builders of #2820 write the multi-draft target graphs and the offload runtime_profiles.

Both use the offload of the ONNX Runtime CUDA execution provider (the open stack microsoft/onnxruntime#33450 → #33622, as for #2784). That runtime has no QMoEOffload operator: it offloads standard com.microsoft::QMoE nodes and is configured with session.moe.offload.* session options. The graphs of this PR keep standard QMoE nodes, and there are two ways to set the options:

• Converter (build_package_offload_v3.py --from-package): the 48 com.microsoft::QMoE nodes of a v12b package (NVFP4 + FP8, custom-op lineage) stay QMoE nodes (only the router_weights input, which the offload backend does not support, is cleared). The decoder session gets session.moe.offload.mode, placement and cpu_execution (--offload-mode, --offload-placement), optional device and host expert budgets, the expert-popularity profile and per-node resident-slot caps (node_overrides), and weights_pool.txt places the routed experts in cpu mmap placement.
• Runtime profiles (build_package_engine_v12b.py --clean-package and build_package_v12d_int4_int8.py --clean-flat-package): the package gets four runtime_profiles that select model_s4.onnx and set the same options. This is the path of the runtime in [Qwen38] Size the QMoE offload budgets from the context tier and run DFlash2 warmup in offload warmup mode #2784, which sizes the offload budgets from the context tier; Add decoder.session_options and host-memory eligibility to RuntimeProfile for portable MoE offload #2762 teaches the runtime to parse the profiles.

Summary of Changes

File	Change
tools/qwen38_graph/build_package_offload_v3.py	New. --from-package rewrites a layered or flat v12b package: model_s2.onnx is the prepared target graph (standard QMoE), model_s3.onnx and model_s4.onnx extend it to 3 and 4 verify rows (build_multidraft_target_onnx, 2 or 3 chained MTP drafts), model.onnx is a copy of model_s4.onnx; weights_pool.txt puts the routed experts on the CPU and the dense weights and the token embedding in host-mapped placement (with --mtp-moe-placement host_mapped, the MTP expert pools that start at an unaligned offset, whatever their pool ids, move to one aligned file next to the source package); genai_config.json gets the session.moe.offload.* options, CUDA-graph capture and kSameAsRequested arena options. --offload-mode required (the default) stops, before anything is written, at a node the runtime would not offload; auto keeps such nodes on the GPU. --target-resident-slots (a per-layer cap) or --expert-profile with --total-resident-slots (--uniform-profile-slots, --min-slots-per-layer) set the resident experts, and a total that the profile cannot fill exactly is refused. The flags that only made sense for the legacy QMoEOffload operator (--migrate-shared-onnx, --out-onnx, --strip-phase, --strip-stats, --target-spare-slots, --dense-offload-qmoe, --dense-stream-min-rows, --dense-stream-wave) are rejected with the reason, and a source graph with QMoEOffload nodes is refused.
tools/qwen38_graph/build_package_engine_v12b.py	--clean-package links model_s2.onnx to model.onnx, generates model_s3.onnx and model_s4.onnx, and writes four runtime_profiles for GPUs with 20 to 34 GiB of device memory. --expert-profile copies an STRP v1 popularity file into the package as expert-profile.bin; without it the builder prints a note.
tools/qwen38_graph/build_package_v12d_int4_int8.py	--clean-flat-package writes the same S=2..4 graphs; the profiles come with the config of --reuse-embed-from.
tools/qwen38_graph/README.md	The offload package section, the multi-draft graphs and the profiles.
test/python/builder/test_qwen38_offload_package_builder.py, _qwen38_qmoe_offload_contract.py	GPU-free tests (290). The contract module mirrors the session-creation checks of the QMoE offload in microsoft/onnxruntime#33622 (commit 1d5ae2f3, test_qmoe_offload_v4.py), so the tests check that every built graph passes them; the tests also create real sessions in the installed onnxruntime.

Notes for review

• The profiles use minimum_effective_host_memory_bytes / maximum_effective_host_memory_bytes and a decoder session_options overlay, so a runtime without Add decoder.session_options and host-memory eligibility to RuntimeProfile for portable MoE offload #2762 rejects the genai_config.json.
• build_multidraft_target_onnx extends a custom-op graph (it needs the FoldMid2 node of qwen.fold). The contrib-operator packages of build_package_contrib_ops.py have no such node, so that builder drops model_s2/s3/s4.onnx and the runtime_profiles, and the rc5 INT4 / INT3 packages carry their own profiles.
• The two flat builders import build_multidraft_target_onnx lazily, so build_package_offload_v3.py has to be in the same folder.
• The package mode of build_package_offload_v3.py expects a v12b-lineage package: model.qwen38_graph with fold.head_r2 and prefill sections and standard QMoE nodes in model.onnx. A source config of an earlier config surface is migrated while it is read (see Add Qwen3.8-Flash graph package builders #2820), and the built genai_config.json only has the keys the reworked Add the qwen38_flash_graph model type and its device decode engine #2727 accepts: fold.head_r2.session_options carries the CUDA-graph defaults and prefill.chunk the chunk, which is checked against kv_capacity before anything is written. A package that already offloads through session options needs no conversion.
• The converter clears the optional router_weights input of the QMoE nodes because the offload backend does not take it; a node that had it then weights the selected experts by router_probs (the count is router_weights_cleared in package_manifest.json).
• The source package is never modified: every graph is saved to a new file, and the model_s2/s3/s4.onnx of a source that already has them are regenerated, not hard-linked.
• The v12d builder no longer carries a branch that looked for a consumer-gpu-24g profile to set qmoe.offload_gpu_layers: it matched nothing, because the profiles are stored as a list.

Testing

• ruff check and ruff format --check pass on the files of this PR; the five builders byte-compile and print --help.
• cd test/python/builder && python -m pytest -q test_qwen38_offload_package_builder.py: 290 passed in about 5 s with the released onnxruntime 1.31.0 CPU wheel that CI installs, no GPU. They build layered and flat packages from synthetic v12b packages and check the output: standard QMoE nodes that pass the modelled session creation of #33622 for each mode (required, auto) and placement (adaptive, static), real session creation in the installed onnxruntime, the source package unchanged, every requested resident-slot total met or refused, every host-mapped MTP pool kept with a defined binding, the config surface, and the rejected legacy flags and graphs. The tests of the fixed defects fail on the earlier builder.
• An earlier revision of the converter (explicit QMoEOffload nodes), run on a copy of a flat NVFP4 + FP8 RC2 package (--vram-limit-gib 32 --expert-profile <expert-profile.bin> --total-resident-slots 960): built in about 3 s, converted 48 QMoE nodes, wrote S=2 / 3 / 4 graphs of 1030 / 1056 / 1082 nodes and a pool of 309 entries (288 on the CPU). The current converter was not run on a real package.
• Not run: loading a package built by these scripts in a CUDA build of [CUDA] Optimize INT4 QMoE offload prefill/decode and S=8 doorbell execution onnxruntime#33622 (the operator contract is checked against its sources at commit 1d5ae2f3, test_qmoe_offload_v4.py L290-L317, and against the released onnxruntime 1.31.0 CPU wheel, which ignores the offload options), and the two clean-package paths (they need the HF checkpoint and the v12b / v12d inputs). The runtime side of the offload is measured with the rc5 packages in [Qwen38] Size the QMoE offload budgets from the context tier and run DFlash2 warmup in offload warmup mode #2784.

Checklist

• Tests added/updated (test_qwen38_offload_package_builder.py; the clean-package paths run against source packages)
• No breaking changes (new file; the two builders only gain output)
• Documentation updated (tools/qwen38_graph/README.md)

[Akshay Sonawane (apsonawane)]

Reviewed all 4 changed files at 3999aef and the named ORT offload/profile contracts. Three new findings are inline. Most importantly, explicit/migration graphs emit legacy QMoEOffload although the claimed ORT #33622 runtime explicitly rejects that schema and requires standard QMoE plus session options. Synthetic Python checks also reproduced dangling nonaligned MTP pool bindings and lost uniform slot totals. No large-package conversion, complete exported-session creation or CUDA budget/performance run was performed. No approval submitted.