Skip to content

bench: give CodSpeed ten walltime samples per project instead of one - #679

Merged
maschwenk merged 1 commit into
mainfrom
max/codspeed-walltime-samples
Oct 22, 2026
Merged

maschwenk merged 1 commit into
mainfrom
max/codspeed-walltime-samples

Conversation

@maschwenk

@maschwenk maschwenk commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

codspeed's walltime number came from one compilation per project, so no stdev, no p-value, and a 10% runner swing reads as a change (#669 is docs only and got "improved by 9.7%")

in walltime mode the bench now runs itself ten more times as child processes and reports those. still one compilation per process, memory and simulation modes untouched

local run with the runner env faked, Compiler-Unions: 10 rounds, mean 75.2 ms, stdev 1.4 ms. about 15 s more per project on the macro runner

compiler_default handed CodSpeed one duration in walltime mode, so there
was no stdev and no p-value. In walltime mode it now re-runs itself ten
times as child processes after its hooked compilation and reports those.
Still one compilation per process; memory and simulation are unchanged.
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Performance change vs main

Workflow run · main 357d1d3ec231 → PR merge 55071077b4b8 (head 9a93d973c0dd)

Clef gate: Running benchmarks because the decision was unavailable (ValueError).

The main and PR release binaries are byte-for-byte identical. No compiler performance difference to measure.

@maschwenk
maschwenk force-pushed the max/codspeed-walltime-samples branch from 3ca62db to 9a93d97 Compare October 22, 2026 02:44
@maschwenk
maschwenk marked this pull request as ready for review October 22, 2026 02:47
@maschwenk
maschwenk enabled auto-merge (squash) October 22, 2026 02:47
@maschwenk
maschwenk merged commit 467182f into main Oct 22, 2026
28 checks passed
@maschwenk
maschwenk deleted the max/codspeed-walltime-samples branch October 22, 2026 02:57
@codspeed

codspeed Bot commented Oct 22, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 10.02%

1 warning

⚠️ 4 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚡ 3 improved benchmarks
✅ 9 untouched benchmarks
⏩ 4 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Walltime drizzle-orm 1.3 s 1.2 s +15.46%
⚡ Walltime webpack 1,019.2 ms 912.2 ms +11.73%
⚡ Walltime Compiler-Unions 647.6 ms 627.3 ms +3.23%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing max/codspeed-walltime-samples (9a93d97) with main (357d1d3)

Open in CodSpeed

Footnotes

  1. 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@maschwenk

Copy link
Copy Markdown
Collaborator Author

oops didn’t mean to merge this without approval. let me know if we need to roll back

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

1 participant