[AMD][dsv4] fix 2P1D eval 503 "all circuits open" (readiness canary + faster CB recovery)#2255
Open
inkcherry wants to merge 1 commit into
Open
[AMD][dsv4] fix 2P1D eval 503 "all circuits open" (readiness canary + faster CB recovery)#2255inkcherry wants to merge 1 commit into
inkcherry wants to merge 1 commit into
Conversation
…recovery Run 28696443568 (dsv4 2P1D gsm8k eval, c=500x512) failed: from request SemiAnalysisAI#1 the router returned 503 "No available prefill workers (all circuits open or unhealthy)" for all 1319 requests and never recovered, so lm_eval produced 0 result files and "Verify eval scores" failed. The launch script declared "Router is ready for benchmarking" as soon as the router /readiness endpoint returned 200 -- which only proves the router process is up, not that it can actually reach a prefill worker and complete a generation. The eval therefore fired into a router whose prefill path was not yet serving, tripped both circuits, and with the default cb-timeout-duration-secs=60 the circuits stayed open past the client retry budget (~31s) so nothing ever recovered. Fix (server-stability / readiness, no eval concurrency lowered, breaker still fully enabled): - Add an end-to-end readiness canary: require ONE successful generation THROUGH the router (POST /v1/chat/completions, max_tokens=1) before "ready for benchmarking", so the eval never starts against a non-serving prefill path. - Speed circuit-breaker recovery (--cb-timeout-duration-secs 15, --retry-max-retries 3) so a transient trip re-closes inside the client retry budget instead of losing the whole eval. Thresholds unchanged. Satisfies MOTIVATION_RULES clause 3 (Fix) via the approved levers "health-endpoint responsiveness" and "circuit-breaker recovery tuning"; no anti-cheat shortcut (concurrency stays 500/512, breaker + score check remain enabled).
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
inkcherry
changed the title
ichbinblau
added a commit
that referenced
this pull request
Jul 21, 2026
- Speed circuit-breaker recovery and gate benchmarking on an end-to-end router readiness canary, so a transient trip or a not-yet-serving router doesn't 503 every request until the client's retry budget is exhausted (port of PR #2255). - Keep decode's admission context-length symmetric with prefill (falling back to prefill's value when unset in models.yaml), so over-length requests are rejected fast by both sides instead of hanging on decode while waiting for a KV transfer prefill never sends (port of PR #2257). Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: inkcherry <mingzhi.liu@amd.com>
adibarra
pushed a commit
that referenced
this pull request
Jul 22, 2026
…ncy sweep (#2293) * [AMD][dsv4] fix 2P1D eval 503s and 8k1k straggler tail - Speed circuit-breaker recovery and gate benchmarking on an end-to-end router readiness canary, so a transient trip or a not-yet-serving router doesn't 503 every request until the client's retry budget is exhausted (port of PR #2255). - Keep decode's admission context-length symmetric with prefill (falling back to prefill's value when unset in models.yaml), so over-length requests are rejected fast by both sides instead of hanging on decode while waiting for a KV transfer prefill never sends (port of PR #2257). Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: inkcherry <mingzhi.liu@amd.com> * add con=500/512 Signed-off-by: Theresa Shan <theresa.shan@amd.com> * set max_total_tokens: 2097152 Signed-off-by: Theresa Shan <theresa.shan@amd.com> Co-authored-by: lixiufei-leo <Xiufei.Li@amd.com> * update config Signed-off-by: Theresa Shan <theresa.shan@amd.com> * [AMD][dsv4] Replace 2P1D DEP8 sweep with 1P1D pure-TP8 in disagg non-MTP search space Drop the 2P1D DEP8 (conc 256/512/1024) and 1P1D DEP8 conc=1024 entries and add a 1P1D pure-TP8 (no EP, no DP-attention) scenario sweeping conc [1, 2, 4, 8, 16, 32, 64, 128] for dsv4-fp4-mi355x-sglang-disagg. Co-authored-by: Cursor <cursoragent@cursor.com> * remove decode context_length Squash of f7295a5 "remove decode context_length", 0be1d30 "Revert 'remove decode context_length' for DeepSeek-V4-Pro", and bfb8988 "remove decode context_length ending with a newline" into a single commit with the same net result. Signed-off-by: Theresa Shan <theresa.shan@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com> * Add perf-changelog.yaml entry for dsv4-fp4-mi355x-sglang-disagg retune Per the automated PR reviewer: any edit to configs/amd-master.yaml requires a paired perf-changelog.yaml entry. Documents the 503/ straggler-tail fixes, the max_total_tokens bump, and the 2P1D DEP8 -> 1P1D DEP8/pure-TP8 concurrency sweep retune from PR #2293. Co-authored-by: Cursor <cursoragent@cursor.com> * Guard router readiness canary with wait_or_die on prefill0_pid The canary poll loop was a plain while/curl loop, unlike every other blocking wait on node rank 0 (container barrier, servers-up barrier, /readiness health barrier), so a prefill crash right after /readiness passed just looked like repeated 503s and burned the full ROUTER_CANARY_TIMEOUT (default 600s) instead of failing fast. Extract the loop into run_router_canary() and run it under wait_or_die so a dead prefill pid aborts it in ~5s like the surrounding barriers. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Signed-off-by: Theresa Shan <theresa.shan@amd.com> Co-authored-by: inkcherry <mingzhi.liu@amd.com> Co-authored-by: lixiufei-leo <Xiufei.Li@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Chun Fang <chun.fang@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On the 2P1D dsv4 gsm8k eval at concurrency 500, every request gets
503 "No available prefill workers (all circuits open or unhealthy)"from the first request and never recovers, so the eval produces no results.server_sglang.shdeclares the router ready as soon as/readinessreturns 200, which only means the router process is up — not that it can actually reach a prefill worker and finish a generation. The eval fires 500 concurrent requests into a prefill path that isn't serving yet, trips both prefill circuit breakers, and the default breaker timeout (60s) is longer than the client retry budget (~31s), so the circuits never reclose.Changed
POST /v1/chat/completions,max_tokens=1) instead of just/readiness.--cb-timeout-duration-secs 15 --retry-max-retries 3). Breaker and health checks are otherwise unchanged; concurrency and the eval itself are untouched.Validated
conc-500 and conc-512 gsm8k both complete,
exact_match≈ 0.958, no 503s, circuits stay closed.cc @Duyi-Wang @billishyahao