[AMD][AgentX] Kimi-K3 FP4 MI355X agentX vLLM - #2403
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # --max-num-batched-tokens and --chunk-size stay at the AMD reference | ||
| # values (4096 / 1024). They were briefly pinned to 768 because LMCache | ||
| # dev HEAD (v0.5.3.dev47) rejects the reference values on this hybrid model: | ||
| # ValueError: Mamba-hybrid models with LMCache require | ||
| # block_size <= max_num_batched_tokens < 2 * block_size ... block_size=768 | ||
| # AssertionError: LMCache chunk size should be a multiple of vLLM block size | ||
| # CONFIRMED on 0.5.1 as well (run 30348060242, g09): the pinned release | ||
| # raises the same ValueError, so these are properties of the LMCache/vLLM | ||
| # Mamba-hybrid integration, not of dev HEAD. Only the LazyMemoryAllocator | ||
| # stall was dev-specific. Both stay pinned to 768 for the LMCache arms. | ||
| # | ||
| # Worth keeping in view either way: at ~106k-token average ISL, 768 would | ||
| # mean ~138 chunked-prefill steps per turn versus ~26 at 4096, so the two | ||
| # settings are not performance-equivalent. | ||
| MAX_NUM_BATCHED_TOKENS="${LMCACHE_MAX_NUM_BATCHED_TOKENS:-768}" | ||
| LMCACHE_K3_CHUNK_SIZE="${LMCACHE_CHUNK_SIZE_OVERRIDE:-768}" | ||
| LMCACHE_CHUNK_SIZE="$LMCACHE_K3_CHUNK_SIZE" | ||
| LMCACHE_CHUNK_SIZE_K27="$LMCACHE_K3_CHUNK_SIZE" | ||
| echo "LMCache: --max-num-batched-tokens=$MAX_NUM_BATCHED_TOKENS --chunk-size=$LMCACHE_K3_CHUNK_SIZE (reference values)" |
There was a problem hiding this comment.
🟡 Two comments in this script mislabel non-reference default values as matching the AMD reference command. (1) The top-of-file knob table says every knob defaults to the reference value and lists GPU_MEM_UTIL as 0.95, but line 640 actually defaults it to 0.88 (a deliberate fleet-specific override). (2) The LMCache branch comment claims --max-num-batched-tokens/--chunk-size 'stay at the AMD reference values (4096 / 1024)', but the same block unconditionally overrides both to 768, and the runtime echo prints '(reference values)' for that 768 setting. Both should be corrected so server.log/CI output and the header docs describe what the script actually does.
Extended reasoning...
This script contains two related but distinct stale-comment issues that mislabel intentional, non-reference overrides as if they matched the AMD reference command.
1. Header knob table (lines 41-50). The top-of-file comment reads: "Perf-search knobs. Each defaults to the reference command's value, so an otherwise-unset run reproduces the reference exactly," followed by a list including GPU_MEM_UTIL 0.95 (reference). But the actual coded default at line 640 is GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.88}", preceded by a 10-line inline comment explaining that 0.88 is a deliberate, measured fleet-specific override (0.95 asks for 273.59 of 287.98 GiB and only cleared 2 of 9 test nodes on cluster:mi355x-amds). Every other knob in that header table (MAX_NUM_SEQS 128, MAX_NUM_BATCHED_TOKENS 4096, LANGUAGE_MODEL_ONLY false, etc.) does match its coded default — GPU_MEM_UTIL is the lone exception, and the header's blanket claim ("an otherwise-unset run reproduces the reference exactly") is therefore false for this knob.
2. LMCache branch comment + runtime echo (lines 223-241). The comment opens with "--max-num-batched-tokens and --chunk-size stay at the AMD reference values (4096 / 1024)." But the same comment block goes on to explain that LMCache's Mamba-hybrid requirement (block_size <= max_num_batched_tokens < 2*block_size) forces both to 768, unconditionally, via MAX_NUM_BATCHED_TOKENS="${LMCACHE_MAX_NUM_BATCHED_TOKENS:-768}" and LMCACHE_K3_CHUNK_SIZE="${LMCACHE_CHUNK_SIZE_OVERRIDE:-768}". The comment is internally self-contradictory (opens by saying "stay at reference," closes by saying "pinned to 768"), and the runtime echo at line 241 compounds it by literally printing LMCache: --max-num-batched-tokens=768 --chunk-size=768 (reference values) — a false label that will appear verbatim in every LMCache run's stdout/server.log.
Why existing code doesn't prevent this: these are plain comments and an echo string; nothing type-checks or lints them against the actual runtime values, so they can drift silently. It reads as leftover text from before the 768 pin became a hard, unconditional requirement (the surrounding comment itself documents that this was "briefly pinned to 768" during dev-HEAD bringup and then confirmed necessary on the pinned release too).
Concrete walkthrough: Run the script with KV_OFFLOAD_BACKEND=lmcache. MAX_NUM_BATCHED_TOKENS and LMCACHE_K3_CHUNK_SIZE both resolve to 768 (their env overrides are unset). Line 241 executes echo "LMCache: --max-num-batched-tokens=768 --chunk-size=768 (reference values)", which lands in server.log/CI stdout claiming reference batching was used, when the actual reference (per the same file's header block) is 4096/1024. Anyone diffing a vllm-simple run's batching config against an LMCache run's server.log would be misled into thinking they're identical when they differ by more than 5x. Similarly, anyone reading the header block in isolation would believe GPU_MEM_UTIL unset means 0.95, when it actually means 0.88.
Impact: purely cosmetic/documentation — the actual runtime values (0.88, 768/768) are correct and intentional; nothing breaks and no wrong flag is passed to vLLM or LMCache. The risk is entirely to a human or downstream tooling reading server.log/CI artifacts or the script header and drawing the wrong conclusion about what was actually run.
Fix: update the header table to list GPU_MEM_UTIL 0.88 (fleet-specific override, not reference), and update the LMCache comment/echo to say the batching values are pinned to 768 for the Mamba-hybrid requirement rather than "reference values."
| additional-settings: | ||
| - "DECODE_NODES=2" | ||
|
|
||
| kimik3-fp4-mi355x-vllm-agentic: |
There was a problem hiding this comment.
🟡 This PR's title and description are English-only, but AGENTS.md explicitly requires bilingual PR titles (<English title> / <中文标题>) and a Chinese translation section (e.g. ## 中文说明) in the body for every PR. Please add the Chinese title suffix and a mirrored Chinese summary section before merge.
Extended reasoning...
AGENTS.md line 7 states an explicit, repo-wide mandate: "PR and GitHub-issue titles & descriptions must be bilingual — include a Simplified Chinese version in addition to English. Title format: <English title> / <中文标题>. In the PR/issue body, follow the English content with its Chinese translation (e.g. a ## 中文说明 section mirroring the summary...). This applies to every PR and every issue..."
PR #2403's title is [AMD][AgentX] Kimi-K3 FP4 MI355X agentX vLLM — English only, with no / <中文标题> suffix. The description body (Summary, Model constraints, Measured results, What is not settled, Other changes, Before merge) is entirely in English, with no ## 中文说明 section or any Chinese translation mirroring it anywhere in the body.
This is a documented, mandatory repo convention (not a subjective style preference invented by a reviewer) — AGENTS.md is the file CLAUDE.md explicitly points to as the source of guidance for this repository, and the requirement is stated unconditionally ("every PR and every issue"). A check that is explicitly requested by repo instructions is squarely in scope for review, even though it concerns PR metadata rather than the code diff itself.
Step-by-step verification:
- Open AGENTS.md line 7 in this repo's checkout — it states the bilingual title/body requirement in absolute terms.
- Open the PR metadata for [AMD][AgentX] Kimi-K3 FP4 MI355X agentX vLLM #2403 — title field is
[AMD][AgentX] Kimi-K3 FP4 MI355X agentX vLLM, no Chinese characters, no/separator followed by a Chinese title. - Search the PR description body for any CJK characters or a
中文说明heading — none are present; the entire body (Summary through Before merge) is English prose and tables. - Conclusion: the PR violates the documented bilingual requirement on both the title and the body.
Impact: this is a process/documentation-compliance gap, not a functional defect — it does not affect the benchmark scripts, the config, or the launcher logic touched by this PR, and nothing will crash or behave incorrectly at runtime as a result. It should still be fixed before merge to keep the repo consistent with its own stated contribution rules, but it does not block correctness of the recipe or the measured results.
Fix: rename the PR title to [AMD][AgentX] Kimi-K3 FP4 MI355X agentX vLLM / <中文标题> and append a ## 中文说明 section to the description that mirrors the Summary (and ideally the key measured-results table) in natural technical Chinese, per the translation-quality bar and terminology table also defined in AGENTS.md.
| kimik3-fp4-mi355x-vllm-agentic: | ||
| image: vllm/vllm-openai-rocm:kimi-k3 | ||
| model: moonshotai/Kimi-K3 |
There was a problem hiding this comment.
🟡 AGENTS.md's 'Adding a benchmark configuration' step requires appending a perf-changelog.yaml entry whenever entries are added to amd-master.yaml, since that file is what triggers benchmark runs — this PR adds kimik3-fp4-mi355x-vllm-agentic plus five diagnostic sibling keys but doesn't touch perf-changelog.yaml. This is a pre-merge housekeeping gap rather than a functional bug: without the changelog entry, none of these configs (including the shippable one) will be picked up for a sweep, so it should be added as part of the 'Before merge' cleanup pass already called out in the PR description.
Extended reasoning...
What's missing: AGENTS.md:140 documents the required process for adding a benchmark configuration: "Add entries to configs/nvidia-master.yaml or amd-master.yaml ..., append to perf-changelog.yaml, then validate with generate_sweep_configs.py full-sweep." AGENTS.md:78 and AGENTS.md:82 reinforce that perf-changelog.yaml is what triggers which configs get benchmarked ("Changes to perf-changelog.yaml trigger benchmark runs"). This PR's diff touches exactly three files — the new .sh script, configs/amd-master.yaml, and runners/launch_mi355x-amds.sh — and perf-changelog.yaml is not among them.
Concrete effect: the new kimik3-fp4-mi355x-vllm-agentic key (and its five diagnostic siblings: -lmprofile, -fp8kv, -retry, -native, -bringup) is added to the master config but is otherwise inert going forward — per the documented mechanism, nothing will cause main to sweep it after merge until a changelog entry is appended. There's direct precedent for the expected pattern: the sister recipe in PR #2371 (kimik3-fp4-b300-vllm-agentic) landed with a matching perf-changelog.yaml append.
Why this isn't blocking: I checked utils/validate_perf_changelog.py, and it only validates the changelog file's own internal consistency (append-only ordering, whitespace preservation, PR-link formatting) — it does not cross-check that every master-config key has a corresponding changelog entry, so CI will not fail on this omission. More importantly, the PR description is explicit that this is unfinished work: full-sweep-enabled is deliberately not set ("the sweep is incomplete"), several open questions are called out under "What is not settled," and the description's own "Before merge" checklist already earmarks a cleanup pass (trimming the diagnostic keys -retry/-bringup/-lmprofile/-native/-fp8kv). Adding the changelog entry naturally belongs in that same finalization pass, alongside deciding which of the six keys actually ship — appending it prematurely, before the diagnostic keys are trimmed, would trigger sweeps against config names that won't exist in the final version.
How to fix: once the diagnostic keys are trimmed and the shippable kimik3-fp4-mi355x-vllm-agentic config is finalized, append a perf-changelog.yaml entry for it (following the - config-keys: [...] format shown in AGENTS.md:154-157), then re-run generate_sweep_configs.py full-sweep to validate.
Step-by-step proof:
AGENTS.md:140states the two-step process for a new benchmark config: (a) add toamd-master.yaml, (b) append toperf-changelog.yaml.- This PR's modified-files list is
benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh(new),configs/amd-master.yaml,runners/launch_mi355x-amds.sh— step (a) happened, step (b) did not. AGENTS.md:78/82establish thatperf-changelog.yaml— notamd-master.yamlalone — is the trigger mechanism read by the sweep-generation tooling on merges to main.- Therefore, immediately after this PR merges, none of the six new
kimik3-*keys will be scheduled for a benchmark run, including the intended-to-shipkimik3-fp4-mi355x-vllm-agentic. - This matches the author's own "Before merge" TODO list (trimming diagnostic keys), so the missing changelog entry is best read as an intentional gap tied to that same pending cleanup rather than an oversight that breaks anything today.
| *) | ||
| echo "Error: unsupported KV_OFFLOAD_BACKEND '${KV_OFFLOAD_BACKEND:-}' (expected: lmcache, lmcache-k27)" >&2 | ||
| exit 1 | ||
| ;; |
There was a problem hiding this comment.
🟡 The default-case error for an unrecognized KV_OFFLOAD_BACKEND only lists lmcache, lmcache-k27 as valid values, but the case statement above it also accepts lmcache-budget, vllm-simple, vllm-simple-fp8, and vllm-native. A user who mistypes a backend name (e.g. vllm-simple2) gets a misleading error that omits most of the actually-valid options.
Extended reasoning...
The bug: kimik3_fp4_mi355x.sh:473-476 is the catch-all *) branch of the KV_OFFLOAD_BACKEND case statement:
*)
echo "Error: unsupported KV_OFFLOAD_BACKEND '${KV_OFFLOAD_BACKEND:-}' (expected: lmcache, lmcache-k27)" >&2
exit 1
;;This fires for any value not matched by an earlier arm. Looking at the full case statement, the actually-valid arms are:
lmcache|lmcache-k27|lmcache-budgetvllm-simple|vllm-simple-fp8vllm-nativemooncake(explicitly rejected with its own dedicated error message, since it's marked unsupported for this model)
So five distinct values pass validation and reach a working configuration (lmcache, lmcache-k27, lmcache-budget, vllm-simple, vllm-simple-fp8, vllm-native — six, excluding mooncake which is a deliberate reject), but the fallback error text names only two of them.
How this manifests: A user (or a config typo, e.g. vllm-simple2 instead of vllm-simple, or native instead of vllm-native) triggers the default branch. The script exits 1 with a message claiming the only valid values are lmcache and lmcache-k27 — actively wrong information that would send someone debugging a typo down the wrong path (e.g. concluding LMCache is the only offload option, when vllm-simple is actually the only backend the PR description says currently works for this model).
Why existing code doesn't catch it: This is a hardcoded string with no relationship to the case arms above it; nothing keeps the two in sync, and there's no test exercising the unsupported-backend path that would catch drift between them.
Step-by-step proof:
- Matrix or user sets
KV_OFFLOAD_BACKEND=vllm-simple2(a typo of the validvllm-simple). agentic_kv_offload_enabledis true, so thecase "${KV_OFFLOAD_BACKEND:-}" instatement runs.vllm-simple2matches none oflmcache|lmcache-k27|lmcache-budget,vllm-simple|vllm-simple-fp8,vllm-native, ormooncake, so it falls to*).- The script prints
Error: unsupported KV_OFFLOAD_BACKEND 'vllm-simple2' (expected: lmcache, lmcache-k27)and exits 1. - The user, trusting this message, tries
lmcache— which per the PR description cannot even serve this model (it hits a Mamba[conv_state, ssm_state]ValueError) — instead of noticing the typo and usingvllm-simple.
Impact: Purely a diagnostic-string bug on an already-failing path; nothing runs incorrectly and no measured/shipped config triggers it (the shipped matrix only ever sets none or vllm-simple). It only misdirects a human during troubleshooting.
Fix: Update the message to list all currently-valid backends, e.g. (expected: lmcache, lmcache-k27, lmcache-budget, vllm-simple, vllm-simple-fp8, vllm-native).
|
any chance yall wanna add dspark to this PR plz @seungrokj #2406 |
|
example here #2367 |
edwingao28
added
full-sweep-fail-fast
agentx-fast
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30495976488 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30496771349 |
b0789a6 to
73e1467
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30503241209 |
|
@hyukjlee the AgentX/AIPerf harness has been updated, please merge origin/main into your branch and refresh your submission. Additional tuning may be necessary depending on the config. I apologize for any inconvenience. This is an automated message. |
…LLM recipe Adds the Kimi-K3 agentic-coding launcher for gfx950, transcribed from vllm-project/recipes models/moonshotai/Kimi-K3.yaml (date_updated 2026-07-25), whose hardware matrix lists mi355x as verified. The default command line is the AMD reference `vllm serve` verbatim -- --trust-remote-code --moe-backend auto --tensor-parallel-size 8 --load-format auto --gpu-memory-utilization 0.95 --mm-encoder-tp-mode data --max-num-seqs 128 --max-num-batched-tokens 4096 --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3, under the amd extra_env block -- plus only the host/port/served-model-name the harness requires. Every perf-search knob (gpu-memory-utilization, max-num-seqs, batched-token budget, AITER a8w4 vs a16w4, --language-model-only, kv-cache-dtype, max-model-len, prefix caching, DSpark) defaults to the reference value, and the optional ones are not emitted at all unless set, so an unset run reproduces the reference exactly. TP=8 is enforced: the MXFP4 checkpoint is 1.561 TB decimal (1.420 TiB, 96 safetensors), ~195 GB/GPU across 8 GPUs of the 288 GB part, so TP=4 cannot load. Upstream strategy_min_gpus agrees (single_node_tp 8, multi_node_dep 16), hence no DP-attention arm. DRAM offload uses LMCache's LMCacheMPConnector against a local `lmcache server`, matching the reference command. LMCache is not present in the kimi-k3 image, so the recipe builds it against ROCm and honours the config's kv-offload-backend version as the build ref; the pinned default carries the --max-gpu-workers/--max-cpu-workers/--l1-align-bytes/--eviction-trigger-watermark CLI the reference passes, which the older --max-workers CLI does not have. L1 is additionally capped to 90% of free /dev/shm, since exceeding it makes LMCache silently disable SHM and fall back to a path that crashes at load. Mooncake is rejected with an explicit error: the upstream recipe marks both kv_store_distributed_mooncake and kv_store_centralized_mooncake unsupported on every hardware target for this model. The none arm is included so the GPU-only KV wall is measured before the host tier can mask it. A measured reference point on this fleet: at GMU 0.90 the pool is 58.08 GiB/GPU = 2,244,655 tokens, i.e. 2.14x concurrency for a 1M-token request. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0.75 was inferred from the reference LMCache command's "size --l1-size-gb to ~75% of the host DRAM you can dedicate" note, but that maps the wrong denominator: dram-utilization scales the machine's total available DRAM, not the portion already earmarked for the pool. 0.80 matches the kimik2.7 agentic key and gives 2399 GB aggregate at TP8, leaving ~460 GB host headroom. The config value is only a budget in any case -- the recipe caps L1 to 90% of free /dev/shm at run time, because exceeding it makes LMCache silently disable SHM and fall back to a path that crashes at load. Comment now says to read the recipe's cap WARNING rather than allocated_cpu_dram_gb when interpreting a result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The LMCache arm cannot initialise without prefix caching. K3's Kimi Delta Attention layers are Mamba KV-cache groups, and vLLM only selects mamba_cache_mode='align' -- the mode that keeps reusable state snapshots -- when prefix caching is enabled. With the reference command's default (prefix caching resolves to False for this model) the connector aborts at KV-transfer init: ValueError: LMCache cannot serve this model's KV cache groups: group 0: MambaSpec with mamba_cache_mode='none' (only 'align' keeps reusable state snapshots) Measured on gfx950 with LMCache v0.5.3.dev47 against vLLM 0.1.dev19253+g5f76ae224. This is a hard dependency rather than a tuning choice, and the agentic matrix has no per-cell env channel to set it from config, so the recipe forces it for the lmcache backend and errors out if the caller explicitly asked for PREFIX_CACHING=false. The none arm is untouched and still reproduces the reference command exactly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ing)
Memory pinning is unavailable in the ROCm container ("CudaPinMemoryBackend:
neither torch cudart nor libcudart is available"), so LMCache disables its
LazyMemoryAllocator and allocates the whole L1 pool before serving anything.
The agentic DRAM budget is therefore an upfront cost, not a ceiling: measured
on gfx950, a 2249 GB pool had still not come up after 178 s, while 512 GB was
healthy in 40 s. 512 GB also matches the AMD reference command.
Caps at LMCACHE_L1_MAX_GB (default 512) and logs a WARNING naming the pool it
actually used, so a result is never read as though it had the full budget.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… GB ceiling Two changes to the LMCache arm. 1. LMCACHE_PROFILE selects the server flag set. 'reference' (default) is the AMD K3 reference command. 'k2.7' reproduces kimik2.7_fp4_mi355x.sh exactly -- --l1-read-ttl-seconds 7200, --chunk-size 256, --max-workers TP*2, --eviction-policy LRU, LMCACHE_BLOCKING_TIMEOUT_SECS=60, and a kv-transfer-config carrying kv_connector_module_path plus the ZMQ-style lmcache.mp.host -- so the one LMCache configuration known to have served this agentic trace on this fleet can be A/B'd without a recipe edit. All of those flags still exist on LMCache dev. 2. The 512 GB L1 ceiling added in the previous commit is removed (default 0 = no ceiling, still capped by /dev/shm). It was not justified: kimik2.7 sizes L1 to TOTAL_CPU_DRAM_GB with only the shm cap and successfully ran a 1199 GB pool on this same cluster, 2.3x the ceiling I imposed. The 178 s timeout that motivated it also extrapolates, from the 40 s/512 GB datapoint, to roughly the time a 2249 GB pool would have needed -- so it evidenced too short a timeout, not an unusable pool size. The agentic README additionally requires consuming TOTAL_CPU_DRAM_GB rather than a model-specific constant. Set LMCACHE_L1_MAX_GB to reimpose a ceiling. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…mba block constraint) Second hard constraint on the LMCache arm, from the same Mamba-hybrid path as the mamba_cache_mode='align' requirement. vLLM enforces block_size <= max_num_batched_tokens < 2 * block_size 'so every prefill step advances exactly one block and every block boundary gets a state snapshot'. K3 resolves to block_size=768, so the reference command's --max-num-batched-tokens 4096 aborts engine init with 'got max_num_batched_tokens=4096, block_size=768. Set --max-num-batched-tokens 768.' Measured on gfx950 with LMCache v0.5.3.dev47. The assignment ignores any inherited MAX_NUM_BATCHED_TOKENS, because both the matrix and the direct driver export the reference 4096, which is invalid here; only an explicit LMCACHE_MAX_NUM_BATCHED_TOKENS overrides. This is a substantive cost of the offload tier on this model, not a formality: at ~106k-token average ISL, a 768-token budget is ~138 chunked-prefill steps per turn versus ~26 at 4096, so LMCache-vs-none is not a like-for-like prefill schedule and must be reported as such. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… c[4,8,16,32] Replaces the placeholder search space with the sweep the measurements point to. GPU-only is limited to low concurrency: at c8 the 1,957,290-token pool already peaks at 98.6% usage with only a 13.8% prefix hit rate, so c8 is past the eviction crossing and anything above it measures thrash. c1/c2/c4 trace the region where the pool still holds the working set. The offload arm is SimpleCPUOffloadConnector. LMCache is dropped entirely -- both its server profiles abort on this model with 'expected a Mamba [conv_state, ssm_state] tensor list, got Tensor' because Kimi Delta Attention supplies one fused state tensor. Measured at c8, vllm-simple lifts the prefix hit rate 13.8% -> 80.7% and drops TTFT from 193 s to 4.8 s for +70% output throughput (15.93 vs 9.37 tok/s), on a heavier ISL sample (129k vs 116k), so c16/c32 are worth probing. c4 locates the crossover below which the GPU pool alone suffices and the offload transfer cost stops paying for itself. gpu-memory-utilization stays at the recipe's 0.88 for every arm including none. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All four vllm-simple cells of run 30370655448 succeeded; all three none cells failed. Two of them (c1, c2) ran on mia1-p01-g11 and died with hard GPU faults -- 21 and 10 hipErrorIllegalAddress respectively, plus NCCL watchdog thread terminations -- after serving 80 and 632 warmup requests successfully. c4 on g16 died with only 'RuntimeError: cancelled', a knock-on from a worker exit. g11 is the same node whose free memory measured 275/212/208 GiB at different times and which hosted the earlier LMCache allocator stall. Adding it to the denylist for hardware faults rather than capacity; the free-memory-driven entries remain unchanged. Retry key set to the three missing none cells at 3600s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--kv-cache-dtype fp8 on top of the winning vllm-simple c8 configuration. ROCm maps fp8 to fp8_e4m3. fp8 halves bytes/token in the GPU pool, which matters here because the pool is the binding constraint: at c8 it peaks at 98.6-99.8% usage and K3 costs ~217 KiB of KV per token, so 1,957,290 tokens should roughly double. Carried on the backend name (vllm-simple-fp8) because the agentic matrix has no per-cell env channel for KV_CACHE_DTYPE. UNVALIDATED on this model: K3's KV pool spans Kimi Delta Attention state and gated-MLA latent, and fp8 support across both spec types is unconfirmed on this build. If it aborts at engine init, that is the answer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every kvnone cell run with --enable-prefix-caching has died during aiperf warmup with hipErrorIllegalAddress and an EngineCore worker killed by signal. This reproduced on three distinct nodes (mia1-p01-g11, -g17, -g19), so it is a kernel bug rather than bad hardware. K3 is hybrid (69 KDA + 24 gated MLA). Enabling prefix caching activates vLLM's Mamba-state block reuse, and the scheduler dump at the fault names exactly that path: new_block_ids=[([1288],[1284],[1630],[1615])] num_computed_tokens=327936 new_block_ids_to_zero=[1615] The only kvnone cell that has ever completed (run 30322098513, c8) predates the flag being defaulted on. The KV-offload arms are unaffected and keep it on -- they pass at 3600s, and LMCache additionally requires it for mamba_cache_mode='align' -- because the connector owns block lifetime. Default is now arm-aware: on when KV_OFFLOAD_BACKEND is set, off otherwise. PREFIX_CACHING=true/false still forces either way for a deliberate A/B. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…efix-caching claim The stock vllm/vllm-openai-rocm:kimi-k3 routes fp8 KV into AITER's mla_gluon decode path, which asserts mla_gluon[bh16bn128] requires batch_size=1, got 128 (run 30411594835). Point the fp8kv key at hyukjleeamd/kimi-k3-mla-b128:latest, which is built with that kernel tuned for batch 128. Also correct the comment added in 836f1f3. It claimed every kvnone cell run with --enable-prefix-caching had died in warmup, and that this made it a kernel bug rather than bad hardware. That was overstated on two data points. In the same run, kvnone c1 on g16 cleared warmup with 0 errors and profiled for 47 minutes with the flag on (12 trajectories, 92.8% server prefix-cache hit) before dying to `srun: error: Node failure on mia1-p01-g16` -- infrastructure, not the kernel. Three cells on three nodes produced three different failures. The default stays off for kvnone to remove a variable while we get a clean baseline, but it is now documented as suspected, pending a matched A/B. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The denylist did not do its job. After excluding g11/g14/g15/g18, run 30412966635 still lost all three kimik3 cells -- to g19 (hipErrorIllegalAddress), g17 (warmup_failure) and g16 (srun node failure). None of those were on the list. Free memory on these nodes moves hour to hour, so a static exclusion mostly shrinks an already-contended pool and lengthens queue time without avoiding the failures it was meant to avoid. Default to no exclusion; set SALLOC_EXCLUDE_NODES explicitly to re-enable it for a specific run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…erving Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ffload arm" Prefix caching is back ON for every arm. Turning it off for kvnone was the wrong call and the evidence never supported it. The trace is built around large shared prefixes -- 98.1% theoretical hit rate, 92.8% measured server-side on a live cell -- so a kvnone run with reuse disabled is not measuring the target workload, and it makes the kvnone-vs-offload comparison at matched concurrency meaningless. The justification for switching it off was two cells (c2/g19, c4/g17, run 30412966635) dying in warmup with the flag on. But c1 in that same run, with the same flag, cleared warmup with 0 errors and profiled 47 minutes clean before dying only to `srun: error: Node failure on mia1-p01-g16`. Weighting two failures over an in-sample counterexample was a mistake, especially in a week where the cluster produced three different failure modes across three nodes. PREFIX_CACHING=false still forces it off for a deliberate A/B. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…V arm Long-context forward passes (~370K tokens with fp8 KV + DRAM offload) exceed vLLM's default 300s VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS, killing the engine with "RPC call to sample_tokens timed out". Default it to 1200s (overridable). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…timeout+mla_gluon patches Isolation A/B for the fp8-KV DRAM-offload TP-rank hang: bf16kvtest keeps DRAM offload (vllm-simple) but drops fp8 KV on the stock image to test whether fp8 KV is the stressor wedging the collective. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
hyukjleeamd/kimi-k3-mla-b128:fp8-kv-fixed replaces :latest for the vllm-simple-fp8 c8 cell. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
g11 is not a free-memory-headroom case like the nodes that made the earlier
broad denylist useless. It fails hard, across unrelated configs:
- kimik3 kvnone c2 (run 30425820793): warmup_failure
- kimik3 vllm-simple-fp8 c8 (run 30428781263): forward passes stalled with
returned=5 / in_flight=2 at zero throughput for 13 minutes, then
EngineCore died
- the kimik2.7 -tune2 cells noted above
Four configs, one node, same outcome. Excluding a single node costs almost
no pool capacity, unlike the g11/g14/g15/g18 list that was rightly turned
off. Kept as ${VAR-default} so SALLOC_EXCLUDE_NODES="" still disables it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ation Isolation variant B: fp8 KV without DRAM offload to test whether the offload path is what wedges the TP collective. Backend is none, so KV_CACHE_DTYPE is forced to fp8 in the recipe; the mla_gluon patch is gated on fp8 KV. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…en stack Run 30442578333 produced the first clean fp8 KV result on Kimi-K3 (c8, 3600s, kvnone, stock image): 32.59 output tok/s, 696 total tok/s/GPU, 155 requests, TTFT 2.6s. What made it work was the recipe patching aiter's mla_gluon kernel from seungrokj's gist plus the widened worker RPC timeout -- not a custom image. Two changes so that result is reproducible and extendable to the offload arm: 1. Pin MLA_GLUON_SRC to gist revision 4b0088c5f, the exact blob that run fetched. The unpinned .../raw/mla_gluon.py URL follows the gist HEAD, so an edit there would swap the decode kernel out from under every fp8 cell with no diff in this repo. 2. Point -fp8kv at the stock vllm/vllm-openai-rocm:kimi-k3 rather than hyukjleeamd/kimi-k3-mla-b128:fp8-kv-fixed. The recipe patches the kernel whenever KV_CACHE_DTYPE=fp8, so stock now reaches the same code path, and this keeps the cell exactly one variable (the SimpleCPUOffloadConnector) away from the run that passed. The custom image remains the fallback: its one trial (30434731504) cleared init and the full warmup, then died at the profiling transition with an async hipErrorIllegalAddress on g14 that was never attributed to a kernel. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fp8 KV is the biggest lever measured on this model. Run 30442578333 (c8, kvnone, 3600s, stock image + patched mla_gluon) scored 32.59 output tok/s and 696 total tok/s/GPU at TTFT 2.6s, against 19.17 / 535 / 69.1s for the bf16 offload cell in the same window. K3 costs ~217 KiB KV/token, roughly 3x K2.7, so halving the KV element size matters more here than it did there. KV_CACHE_DTYPE now defaults to fp8 for every arm, set before the KV_OFFLOAD_BACKEND case. That placement is required, not cosmetic: the agentic matrix has no per-cell env channel, so a kvnone cell has no other way to request fp8, and the mla_gluon patch gates on KV_CACHE_DTYPE=fp8 and has to see it already set. KV_CACHE_DTYPE=auto still gives a bf16 A/B. Adds -fp8sweep: kvnone c[1,4,8] plus vllm-simple-fp8 c[8,16]. The kvnone c8 cell repeats 30442578333 as a control; c1/c4 replace the bf16 points (159 and 421 tok/s/GPU, run 30430688459) so the curve is one dtype throughout. The offload cells isolate what SimpleCPUOffloadConnector costs under fp8, and c16 probes whether fp8 moves the ceiling past the bf16 collapse there. Prefix caching stays on for every arm, unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…dation `kv-offload-backend: none` parses as the STRING 'none', which pydantic rejects against KVOffloadBackendMetadata. validate_master_config validates the whole file, so this one cell was failing every dispatch off this branch, not just the fp8kvtest key (run 30453389265, get-jobs). The arm wants no offload at all, which is `kv-offloading: none` with no backend key -- the form on amd/agentx_k3_vllm_0729 that run 30442578333 actually executed. The cherry-pick into this branch (cbb505d) changed it to kv-offloading: dram plus a string backend, which never validated here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…8 c8) Retries the two cells that died in run 30453589555 and adds none c16. Both deaths were GPU faults (hipErrorUnknown), not timeouts. No guard was close to firing: VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1200 against a ~180s stall, warmup-grace-period 1800 against 420s elapsed, and the only "timeout" strings in the server log are source lines inside the traceback. A timeout would also not surface as hipErrorUnknown. GPU faults have hit every arm on this cluster today -- bf16 kvnone on g11/g17/g18/g19, fp8 kvnone on g10, fp8 + offload connector on g11 and g19 -- so both cells are retried unchanged rather than "fixed". The offload cell was measurably slower before it died (2.7x longer to 5 completions than the no-connector run at the same concurrency), but slowness did not kill it, and that is a throughput question to settle with a number rather than a config change. none c16 is new. The no-offload arm has never been measured above c8, and fp8 roughly halves KV bytes per token, so the concurrency ceiling should have moved; bf16 collapsed at c16 (588 tok/s/GPU, TTFT 295s) and this checks whether fp8 changes that. Also corrects the KV_CACHE_DTYPE header comment, which still claimed a default of auto after acd6d30 made fp8 the default. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add the MI355X DSpark wrapper, TP8 vLLM configuration, synthetic golden-AL handling, and performance changelog entry. 中文：新增 MI355X Kimi-K3 DSpark AgentX 包装脚本与 TP8 vLLM 配置，接入合成黄金接受长度，并登记性能变更日志。
fp8 KV + SimpleCPUOffloadConnector has now failed 3/3 (runs 30453589555, 30457683686; nodes g17/g19/g12), every time a GPU fault at under 10% pool usage. The same connector on bf16 (run 30431115226) ran to 99.9% with zero faults, and fp8 without the connector passes at c4/c8/c16. The two runs are config-identical except kv_cache_dtype -- same vLLM g5f76ae224, GMU 0.88, max-num-seqs 128, same cpu_bytes_to_use_per_rank -- and diffing the two vllm_command.txt files leaves --kv-transfer-config as the only delta. Both allocate an identical 14122 CPU blocks while GPU token capacity goes 1,957,290 -> 3,859,490. manager.py:199 scales num_cpu_blocks with num_gpu_blocks, so the block count held and tokens/block doubled: fp8 takes a 128-token page where bf16 takes 64. A 128 page routes MLA into aiter's Gluon b128 kernel, which fp8 survives on its own but has not survived alongside block-granular connector copies. Add KV_BLOCK_SIZE (-> --block-size) and a vllm-simple-fp8-b64 backend that pins it to 64, plus a one-cell fp8b64 key. Keeps fp8 and moves exactly one variable off the failing config. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
run_eval defaults agentic scenarios to swebench (benchmark_lib.sh:1464), but EVAL_FRAMEWORK wins over that default at :1471. Export it as lm-eval the way minimaxm3_fp4_mi355x.sh does, left overridable so EVAL_FRAMEWORK=swebench can still reach the SWE-bench path this recipe already carries. The eval arm runs the one config proven on this stack: fp8 KV, no offload, c8 -- the control from run 30442578333 (696 tok/s/GPU), repeated clean in 30453589555. The offload arm is deliberately excluded; it is 3/3 dead under fp8 and is being isolated separately by the fp8b64 key. Dispatch with --evals-only: mark_eval_entries flips single-node agentic rows to run-eval=True (generate_sweep_configs.py:360), yielding one eval-only cell. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ert on K3 The block-size attempt (run 30505656455) is void. K3 is hybrid, so vLLM raises the attention block size until the attention page is at least the mamba/KDA page (interface.py:911) and silently overrode --block-size 64 to 1536. Measured pages are bf16 768 / fp8 1536, so that cell was byte-identical to the earlier eager failures and died the same way. Keep the KV_BLOCK_SIZE knob but document it as inert here. This also explains the connector's identical 14122 CPU blocks across dtypes: fp8 halves bytes/token but doubles page tokens, so page bytes -- and manager.py:199's pool -- are unchanged. New lever: lazy offload. Eager stores every completed block to CPU at once; every fp8+connector death has come while the GPU pool was nearly empty (11.6% usage, 0.0% external hit rate in 30505656455), i.e. mid store-storm with nothing yet to gain from the host tier, and the last two were a 600s RCCL _ALLGATHER_BASE hang rather than an OOM. Lazy only stores under GPU-block pressure, removing nearly all connector traffic in that regime. It is also the mode kimik3_fp4_mi355x_mtp.sh already pins for gfx950 (PR #2367). Add a vllm-simple-fp8-lazy backend and swap the fp8b64 key for fp8lazy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Use the exact GPU-only c1 DSpark serving defaults requested for MI355X: block rejection, 128 sequences, 4096 batched tokens, 0.95 GPU memory utilization, multimodal data parallelism, and no eager or prefix-cache override. Add regression coverage for the wrapper and generated matrix. 中文：将 MI355X Kimi-K3 DSpark 对齐到指定的 ROCm GPU-only c1 基线：使用 block rejection、128 个序列、4096 个批处理 token、0.95 GPU 显存利用率、multimodal data 并行，并移除 eager 与显式 prefix-cache 覆盖。同时为 wrapper 和生成矩阵添加回归测试。
…armup Run 30511545046 tried lazy alone and is uninterpretable. It landed just after #2415 swapped the agentic warmup from --agentic-cache-warmup-duration 600 to --warmup-requests-per-lane 10 (benchmark_lib.sh:1793, plus a new utils/aiperf pin), so it moved two variables at once. Its "0 successful / 49 total, OSL=1" is the new warmup's one-token lane primers, not a lazy-offload symptom -- the cell never left warmup. That also invalidates the seven earlier fp8+offload failures as baselines: all of them ran the old time-based warmup. So run the eager control at c8 on the SAME warmup as lazy, in one dispatch, and compare those two only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Raise the recipe's GPU_MEM_UTIL default 0.88 -> 0.90 and add kimik3-fp4-mi355x-vllm-agentic-fp8none-gmu90, a single GPU-only fp8-KV concurrency curve. The agentic matrix has no per-cell env channel, so the memory fraction has to move on the recipe default. The offload arm is left out on purpose: it has failed 4/4 under fp8, and mixing it into this dispatch would put failure-prone cells alongside the control curve. c16 is the new point -- the no-offload arm has never been measured above c8. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Match the same-image vLLM reproducer that initializes successfully by lowering the serving max sequence slots from 128 to 16. Keep the eval limit and all other DSpark settings unchanged for a one-variable validation. 中文：将服务端最大序列槽位从 128 降至 vLLM 在同一镜像中验证可正常初始化的 16。为保证单变量验证，评估上限及其他 DSpark 配置均保持不变。
95257ad to
f2feeb8
Compare
0.90 does not fail at engine init -- it comes up clean with a 4,302,357-token pool and then dies mid-prefill. Run 30521327099 lost c4, c8 and c16 on three separate nodes with one signature: HSA_STATUS_ERROR_OUT_OF_RESOURCES / "Available Free mem : 0 MB" on 4 of 8 ranks, surfacing as hipErrorUnknown at gpu_model_runner.py:314. Every death was at num_computed_tokens 360,960 to 364,032, num_running_reqs=1, kv_cache_usage ~21%. Not concurrency, not KV exhaustion: the transient chunked-prefill workspace outgrows the ~22 GiB of unreserved headroom 0.90 leaves. 0.88 leaves ~28 GiB and served 987 requests at ISL p90 374,550 / p95 511,146 with a 0/165 error rate (run 30453589555), on a serve command byte-identical apart from this flag. The extra pool 0.90 buys is unusable anyway -- peak usage at death was 21%. Key renamed fp8none-gmu90 -> fp8none; it never produced a result at 0.90. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
LMCache now supports Kimi-K3 upstream. Two dev commits are what make it work:
#4206 (07-23) _MambaUnifiedViewEdit -- re-views a Mamba/KDA UNIFIED state
tensor as a single attention tensor. This is the fix for the blocker
that has killed every LMCache attempt on this model:
"ValueError: expected a Mamba [conv_state, ssm_state] tensor list,
got Tensor" -- K3's Kimi Delta Attention supplies one fused tensor.
#4278 (07-28) _SubpagedMLAAttentionViewEdit -- re-views K3's kernel-paged
MLA tensor [N*12, 64, 576] as logical pages [N, 768, 576].
LMCache also shipped a validated K3 recipe (#4277). Neither fix is in 0.5.1
(this recipe's pin) nor in any PyPI release (latest 0.5.2), so the recipe now
builds from source at the #4278 merge commit with ROCm flags; nightly wheels
are CUDA-only.
Correctness changes beyond the version bump:
- --mamba-cache-mode align passed explicitly. It is the only mode the KDA
backend supports and the only one that keeps reusable state snapshots.
- VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1, required by upstream's K3 recipe.
- chunk-size and max-num-batched-tokens now DERIVE from vLLM's unified block
size N instead of being hardcoded to 768. N is not constant: upstream
quotes 768 for K3 at TP8, but that is bf16. Under this recipe's default
fp8 KV, three runs (30453589555, 30521327099, 30525206671) all log
"Setting attention block size to 1536 tokens". Hardcoding 768 violated both
LMCache constraints (chunk_size % N == 0, N <= mnbt < 2N) on every default
run. Now 1536/3000 under fp8, 768/1464 under bf16, both range-checked.
- New lmcache-k3 backend name selects upstream's minimal validated server
command; the `reference` profile's extra flags were never validated on K3.
- The install asserts both edit classes are registered before the 1.56 TB
weight load, so a stale build fails in seconds rather than 20 minutes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Attempt 1 (run 30559676068) cleared every LMCache-specific hurdle: the ROCm source build at #4278, both _MambaUnifiedViewEdit and _SubpagedMLAAttentionViewEdit registered, --mamba-cache-mode align accepted, N=1536 confirmed in the startup log, LMCache server healthy in 11s. The long-standing "expected a Mamba [conv_state, ssm_state] tensor list, got Tensor" blocker is gone. It then died on the allocation wall the no-offload arm already has: HSA_STATUS_ERROR_OUT_OF_RESOURCES / "Available Free mem : 0 MB". But at num_computed_tokens=184,320 and kv_cache_usage 5.2%, versus 360,960-373,248 at 10-21% without the connector -- the threshold roughly halved, because LMCacheMPConnector holds GPU-side staging buffers on top of the transient chunked-prefill workspace. Two levers, scoped to the LMCache arm so the no-offload curve is untouched: - max_num_seqs 128 -> 32. The chunked MLA workspace is sized from max_num_seqs and had already doubled under fp8 (98,304 -> 196,608 rows). 128 slots is meaningless at concurrency 8; the widest tree fan-out in the trace is 13 live streams, so 32 keeps headroom and cuts the workspace 4x. - gpu_memory_utilization 0.88 -> 0.85 (~36 GiB free vs ~28). An arm with an extra GPU-side consumer needs more slack than one without. Both overridable; both should be walked back toward the no-offload values before quoting a like-for-like number against 682.63 tok/s/GPU. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
5 participants
Summary
Kimi-K3 FP4 (MXFP4) single-node agentic-coding trace benchmark on MI355X with vLLM (TP8).
benchmarks/single_node/agentic/kimik3_fp4_mi355x.sh— KV tiersnone(GPU-only) andvllm-simple(SimpleCPUOffloadConnector), plusvllm-native(OffloadingConnector) and LMCache variants behind flags.kimik3-fp4-mi355x-vllm-agentic, dram-utilization 0.80, 3600s.vllm/vllm-openai-rocm:kimi-k3, modelmoonshotai/Kimi-K3.full-sweep-enabledis deliberately not set — the sweep is incomplete. See "What is not settled" below. This PR lands the recipe and the config so the work is reviewable and reproducible; it does not claim a best config.Model constraints found during bringup
gpu-memory-utilization0.88, not the upstream recipe's 0.95. 0.95 asks for 273.59 of 287.98 GiB and cleared only 2 of 9 cells on this fleet — most nodes carry more than 14.4 GiB of overhead, and free memory varies hour to hour.[conv_state, ssm_state]list and KDA supplies one fused tensor, so it fails withValueError: expected a Mamba [conv_state, ssm_state] tensor list, got Tensor. Present inlmcache==0.5.1, not a dev-branch regression.vllm-simpleis the only working offload tier. The LMCache keys are kept only to document the dead end and should be trimmed before merge.block_size <= max-num-batched-tokens < 2*block_size(block_size 768) andchunk_size % 768 == 0; the recipe applies these automatically.Measured:
vllm-simpleconcurrency curve (3600s, GMU 0.88, prefix caching on)From run 30370655448, aggregated artifacts:
c8 is the peak. Past it the host tier stops keeping up: c16 loses 36% of per-GPU throughput and TTFT jumps from 3.9 s to 295 s, c32 worse again. This trace is extremely prefix-heavy (ISL p50 ~167k tokens), so throughput is dominated by input, and once queueing starts the collapse is steep.
K3 costs roughly 217 KiB of KV per token versus K2.7's ~69 KiB, and the 1.42 TiB of weights leave only ~18% of HBM for KV — about 52 GiB/GPU, a 1.96M-token pool.
What is not settled
No valid
nonedata at 3600s. Every attempt has died, on six different nodes and in three different ways:hipErrorIllegalAddress(g11, g19),warmup_failure(g17), andsrun: error: Node failure(g16) — plus two runs lost to my own concurrent dispatches preempting them. The onlynonecell that ever completed was c8 at 900s / GMU 0.95 / no prefix caching, which is not comparable to the table above. Thenonec1/c2/c4 cells in the shipped config are therefore unvalidated at 3600s.The c4 head-to-head (
nonevsvllm-simple) is unresolved, and cannot be read off this PR: the current default runsnonewith prefix caching off andvllm-simplewith it on, so a matched pair is still needed.The
nonearm hits an intermittenthipErrorIllegalAddressthat looks like a vLLM bug on K3's hybrid Mamba path. Prefix caching stays ON everywhere (the trace is 98.1% shared-prefix; a run without reuse does not measure the workload, and LMCache requires it). But every crash carriesnew_block_ids_to_zeroin the scheduler dump — the Mamba-state block-reuse path that only exists when prefix caching is on:hipErrorIllegalAddressnew_block_ids_to_zero=[943], 327,936 tokhipErrorIllegalAddressnew_block_ids_to_zero=[1190], 121,344 tokhipErrorIllegalAddressnew_block_ids_to_zero=[1615], 327,936 tokwarmup_failuresrunnode failureFour crashes on four nodes, one clean counterexample, so it is intermittent rather than deterministic. It is not context-length driven: 121K and 328K computed tokens both fault, at 6.6% KV utilisation. This needs an upstream fix, not a benchmark workaround.
PREFIX_CACHING=falseexists for a deliberate A/B but is not the shipped default.fp8 KV cache is blocked, but the blocker has moved. On the stock image,
--kv-cache-dtype fp8routes decode into AITER'smla_gluon, which assertsmla_gluon[bh16bn128] requires batch_size=1, got 128. A b128-tuned image (hyukjleeamd/kimi-k3-mla-b128) clears that assertion and delivers the expected capacity win — GPU KV 3,859,490 tokens vs 1,957,290 on bf16, +97%. But both test cells then hung:TimeoutError: RPC call to sample_tokens timed out, with every worker last logging an untuned[aiter] M:196608, N:3072, K:512 ... using torch solution:0GEMM (M = 3,072 scheduled tokens x 64 KV heads). Reproduced identically on two nodes (g17, run 30420218964; g09, run 30422331114), so it is deterministic in the fp8 path rather than environmental. fp8 is not included here.Other changes
runners/launch_mi355x-amds.sh: the salloc denylist is now off by default. It did not work — after excluding g11/g14/g15/g18, a full run still lost all three cells to g19, g17 and g16, none on the list. It was shrinking an already-contended pool and lengthening queue time without preventing the failures.SALLOC_EXCLUDE_NODESstill re-enables it per run.Before merge
Trim the diagnostic config keys (
-retry,-bringup,-lmprofile,-native,-fp8kv) — they exist to drive individual probe runs and are not intended to ship.