Skip to content

test(deepseek4): CPU-parity test for HC pre-mix kernels#510

Merged
davide221 merged 4 commits into
Luce-Org:mainfrom
cheese-cakee:test/ds4-hc-cpu-parity
Jul 11, 2026
Merged

test(deepseek4): CPU-parity test for HC pre-mix kernels#510
davide221 merged 4 commits into
Luce-Org:mainfrom
cheese-cakee:test/ds4-hc-cpu-parity

Conversation

@cheese-cakee

@cheese-cakee cheese-cakee commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

What

Adds a CPU-parity unit test for the DeepSeek-V4 Hierarchical-Controller pre-mix kernels (server/src/deepseek4/deepseek4_hc_cuda.cu), registered as the deepseek4_hc_cuda ctest. HIP-only, no runtime code touched.

Why

The HC pre step is hand-written CUDA/HIP with block reductions whose width follows the wavefront, and it had no correctness backstop — the sibling test_rms_norm_hip covers only RMSNorm. This guards the HC path (mix matvec + NHC=4 Sinkhorn + gated combine) against reduction and wave-width regressions on AMD the same way the RMSNorm test does: it compares the GPU kernels against a CPU reference on the device's actual wavefront width, so wave32 cards (gfx1100 / gfx1151) and any future wave64 card each validate the mode they run.

How

  • server/test/test_deepseek4_hc_cuda.cpp drives deepseek4_cuda_hc_pre with the DS4-Flash config (n_hc = 4, so mix_dim = 24; n_embd = 512 so hc_dim = 2048 spans several strided load iterations in both reductions).
  • The CPU reference reproduces both device kernels exactly:
    • hc_mix_norm_kernel — per-row RMS-normalized dot fn[row] . state.
    • hc_finish_kernel — sigmoid pre/post gates, per-dst row-softmax followed by the sinkhorn_iters normalization sweeps, then the gated combine working[d] = sum_h pre[h] * state[h][d].
  • fn is round-tripped through __half on the host so both sides consume identical weights; the check covers all three outputs (working, post, comb).
  • The CMake target mirrors the existing test_rms_norm_hip block (LANGUAGE HIP, includes src + hip_compat, links dflash_common) and prints the device warpSize so the run doubles as a wave-width report.

Verification

Compiled the test together with its kernel translation unit directly (hipcc --offload-arch=gfx1151) and ran on gfx1151 (Radeon 8060S, wave32):

[ds4-hc-test] device=Radeon 8060S Graphics warpSize=32 n_hc=4 n_embd=512 sinkhorn_iters=4
[ds4-hc-test]   working  max_abs=1.490e-07
[ds4-hc-test]   post     max_abs=1.192e-07
[ds4-hc-test]   comb     max_abs=8.941e-08
[ds4-hc-test] max_abs_diff=1.490e-07 max_rel_diff=1.717e-05
[ds4-hc-test] PASS

Tolerance is 1e-4 (matching test_rms_norm_hip), ~3 orders of magnitude above the observed margin, leaving headroom for wave64 reduction ordering. The deepseek4_hc_cuda CMake target is wired verbatim from the merged test_rms_norm_hip target; a full CMake build of the test was not run in this pass.

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Re-trigger cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread server/test/test_deepseek4_hc_cuda.cpp Outdated
cheese-cakee and others added 2 commits July 11, 2026 17:06
deepseek4_cuda_hc_pre_mix is called on the split mix plus CPU-finish decode
path, so the 'currently uncalled anywhere in server/' note was wrong. Describe
it as a separate mix-only entry instead.
@davide221 davide221 merged commit 0e00236 into Luce-Org:main Jul 11, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants