[experimental][AgentX]: B300 deterministic AIPerf - #2408
Conversation
中文：将 AIPerf 子模块固定到 PR #26 的测试分支，并将 AgentX 预热改为每条并发通道 10 个请求。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
中文：为 Kimi K3 B300 的每通道 10 请求确定性预热实验触发完整扫描。
cquil11
changed the title
cquil11
changed the title
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # utils/aiperf/docs/tutorials/agentx-mvp.md. | ||
| local result_dir="$1" | ||
| local duration="$DURATION" | ||
| local cache_warmup_duration="${AIPERF_AGENTIC_CACHE_WARMUP_DURATION:-600}" | ||
| local warmup_requests_per_lane="${AIPERF_WARMUP_REQUESTS_PER_LANE:-10}" | ||
|
|
||
| # Fast mode is an e2e-only feedback preset used before canonical one-hour | ||
| # sweeps. AIPerf already exposes both controls, so no AIPerf patch is | ||
| # required. | ||
| if [[ "${AIPERF_EXPERIMENTAL_FAST:-0}" == "1" ]]; then | ||
| duration=1200 | ||
| cache_warmup_duration=300 | ||
| fi | ||
|
|
||
| export AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES="${AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES:-0}" |
There was a problem hiding this comment.
🟡 Fast mode (AIPERF_EXPERIMENTAL_FAST=1) previously halved cache warmup along with duration, but this PR removes that override without adding an equivalent reduction for warmup_requests_per_lane — fast mode now only shortens --benchmark-duration while warmup stays at the canonical 10 requests/lane. This also leaves AGENTS.md:122 stale, since it still describes the old duration-based warmup ('5-minute cache warmup ... canonical AgentX timing remains a 10-minute cache warmup') that no longer exists after this change.
Extended reasoning...
What changed and what broke: build_replay_cmd() in benchmarks/benchmark_lib.sh swaps the duration-based AgentX cache warmup (--agentic-cache-warmup-duration) for a deterministic request-count warmup (--warmup-requests-per-lane). Before this PR, the AIPERF_EXPERIMENTAL_FAST=1 branch shortened both knobs for fast e2e feedback runs:
if [[ "${AIPERF_EXPERIMENTAL_FAST:-0}" == "1" ]]; then
duration=1200
cache_warmup_duration=300 # <-- halved from the 600s default
fiAfter the PR, only duration is shortened; the cache_warmup_duration=300 line was deleted with no analogous adjustment to warmup_requests_per_lane:
if [[ "${AIPERF_EXPERIMENTAL_FAST:-0}" == "1" ]]; then
duration=1200
fiStep-by-step proof:
- A user dispatches an AgentX preflight with
agentx-fast=true(per AGENTS.md:122), which setsAIPERF_EXPERIMENTAL_FAST=1. build_replay_cmd()runs;durationbecomes1200(20 min), butwarmup_requests_per_lanestays at its default of10(AIPERF_WARMUP_REQUESTS_PER_LANE:-10}), identical to a canonical one-hour run.- The emitted aiperf command carries
--benchmark-duration 1200 --warmup-requests-per-lane 10— the same warmup cost as a full canonical sweep, just with a shorter profiling window afterward. - Anyone relying on fast mode to also cut warmup time (as it did pre-PR) gets a smaller-than-expected speedup, and doesn't discover this unless they read the new code — nothing signals the behavior change.
Why nothing catches this: there's no test or assertion tying fast-mode env vars to a proportional warmup reduction; it was purely convention in the old code. bash -n and git diff --check (the PR's stated validation) only check syntax, not behavior parity with the prior fast-mode contract.
Documentation impact: AGENTS.md:122 still reads: "This selects a 5-minute cache warmup and 20-minute profile for every AgentX job... canonical AgentX timing remains a 10-minute cache warmup and 1-hour profile." That text describes the removed --agentic-cache-warmup-duration mechanism (5 min / 10 min splits) which no longer exists in either mode — warmup is now request-count based via --warmup-requests-per-lane, and both fast and canonical runs use the same count (10 requests/lane). This line should be rewritten to describe the new request-count semantics and either state that fast mode currently does not shorten warmup, or add a corresponding fast-mode override for warmup_requests_per_lane if a shorter warmup is still desired.
Fix options: (a) add an explicit fast-mode override, e.g. warmup_requests_per_lane=... inside the AIPERF_EXPERIMENTAL_FAST branch to restore proportional warmup savings, and/or (b) update AGENTS.md:122 to reflect that warmup is now request-count based rather than duration based, removing the now-inaccurate "5-minute cache warmup" / "10-minute cache warmup" wording.
This is a defensible, low-risk change on its own (10 request-per-lane warmup with no idle delay is already fast), so nothing crashes or produces incorrect benchmark data — it's a silent behavior change to an experimental fast-mode preset plus a stale doc, not a functional regression.
| # utils/aiperf/docs/tutorials/agentx-mvp.md. | ||
| local result_dir="$1" | ||
| local duration="$DURATION" | ||
| local cache_warmup_duration="${AIPERF_AGENTIC_CACHE_WARMUP_DURATION:-600}" | ||
| local warmup_requests_per_lane="${AIPERF_WARMUP_REQUESTS_PER_LANE:-10}" |
There was a problem hiding this comment.
🟡 The AIPerf warmup rename (AIPERF_AGENTIC_CACHE_WARMUP_DURATION → AIPERF_WARMUP_REQUESTS_PER_LANE) in build_replay_cmd() isn't propagated to its consumers: the 4 GB300 DeepSeek-V4 agentic recipe YAMLs (disagg-gb300-1p1d-dep4-dep8-c128-mtp-agentic.yaml:201, agg-gb300-tp8-mtp-agentic.yaml:147, agg-gb300-tp4-mtp-agentic.yaml:144, disagg-gb300-1p1d-dep8-dep8-c384-mtp-agentic.yaml:202) still set the now-dead AIPERF_AGENTIC_CACHE_WARMUP_DURATION: "600", and benchmarks/multi_node/amd_utils/server_sglang.sh:989 still forwards that dead var into the client container instead of the new AIPERF_WARMUP_REQUESTS_PER_LANE. Nothing crashes since those recipes just fall back to the new default (10 requests/lane), but the stale env vars are misleading and worth cleaning up, and AMD multi-node runs currently have no way to override the new knob via that allowlist.
Extended reasoning...
build_replay_cmd() in benchmarks/benchmark_lib.sh (used by every agentic recipe, including the 4 GB300 DeepSeek-V4 ones despite the PR's 'B300-only' framing) is changed to read AIPERF_WARMUP_REQUESTS_PER_LANE (default 10) and emit --warmup-requests-per-lane, replacing the old AIPERF_AGENTIC_CACHE_WARMUP_DURATION (default 600) / --agentic-cache-warmup-duration path entirely. That is exactly the intended behavior change described in the PR summary, so this itself isn't a bug — but two consumers of the old var were left un-migrated.
Step-by-step proof:
- Before this PR,
disagg-gb300-1p1d-dep4-dep8-c128-mtp-agentic.yaml:201(and the other 3 GB300 agentic recipes) setAIPERF_AGENTIC_CACHE_WARMUP_DURATION: "600"in their env, whichbuild_replay_cmd()read via${AIPERF_AGENTIC_CACHE_WARMUP_DURATION:-600}and passed as--agentic-cache-warmup-duration 600. - After this PR,
build_replay_cmd()no longer references that var at all — grep confirms zero remaining reads inbenchmark_lib.sh. The recipe'sAIPERF_AGENTIC_CACHE_WARMUP_DURATION: "600"setting becomes a silent no-op: it's still exported into the environment but nothing consumes it. - Instead,
build_replay_cmd()now readsAIPERF_WARMUP_REQUESTS_PER_LANE, which none of these 4 YAMLs set, so they silently pick up the new default of10requests/lane via${AIPERF_WARMUP_REQUESTS_PER_LANE:-10}. - Separately,
benchmarks/multi_node/amd_utils/server_sglang.sh:989forwardsAIPERF_AGENTIC_CACHE_WARMUP_DURATION(now dead) into the client container's env allowlist, but does not forwardAIPERF_WARMUP_REQUESTS_PER_LANE. So even if an AMD multi-node recipe wanted to override the new per-lane count, there's currently no path for that override to reach the client container.
Existing code doesn't catch this because there's no validation that env vars referenced in recipe YAMLs are actually consumed anywhere — a renamed/removed var is simply silently ignored rather than erroring, and the env-forward allowlist is a static list that has to be manually kept in sync with whatever knobs build_replay_cmd() currently exposes.
Impact is limited: because the old default was also 600s (i.e., these recipes were never overriding the default, just re-stating it), no customized behavior is actually being lost — they're just moving to the new mechanism's default like every other agentic recipe would after this PR merges, which is the PR's intended outcome. Nothing errors or crashes. The concrete cost is (a) 4 lines of dead/misleading config in production recipe YAMLs that should be updated or removed, and (b) a gap in the AMD env-forward allowlist that blocks overriding the new knob for multi-node AMD runs until it's added.
Fix: replace AIPERF_AGENTIC_CACHE_WARMUP_DURATION: "600" with the equivalent AIPERF_WARMUP_REQUESTS_PER_LANE setting (if a non-default value is actually desired) or drop the stale key entirely in the 4 YAMLs, and swap the forwarded var name at server_sglang.sh:989 from AIPERF_AGENTIC_CACHE_WARMUP_DURATION to AIPERF_WARMUP_REQUESTS_PER_LANE.
Point the experimental B300 sweep at the deterministic warmup implementation rebuilt directly on the AgentX v1.0 branch. 中文：将实验性 B300 扫描指向直接基于 AgentX v1.0 分支重建的确定性预热实现。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30467242205 |
Pin the AIPerf submodule to the deterministic warmup branch based on current fork main. Enable headless realtime metrics at a fixed 30-second cadence for sweep visibility. 中文：将 AIPerf 子模块固定到基于当前 fork main 的确定性预热分支，并启用每 30 秒输出一次的无界面实时指标，便于观察扫描运行状态。 Signed-off-by: Cam Quilici <cjquilici@gmail.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30471975130 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30475706309 |
|
@cquil11 |
|
/stage-results 30475706309 |
|
@cquil11 staged run 30475706309: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-07-29~r30475706309 This shared staging slot remains available until the next |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30486887478 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30475706309 |
|
/stage-results 30475706309 |
|
@cquil11 staged run 30475706309: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-07-29~r30475706309 This run remains available across future @cquil11 已将运行 30475706309 发布到预发布环境：https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-07-29~r30475706309 后续的 |
1 participant
Summary
Experimental B300-only test:
utils/aiperfto feat: add per-lane request budgets to agentic cache warmup aiperf#26 at3d77dd34566c9d312c21fb7a96fcb39d803a7f07--warmup-requests-per-lane 10Validation
bash -n benchmarks/benchmark_lib.shgit diff --check