Repository navigation
bench: give CodSpeed ten walltime samples per project instead of one - #679
Conversation
compiler_default handed CodSpeed one duration in walltime mode, so there was no stdev and no p-value. In walltime mode it now re-runs itself ten times as child processes after its hooked compilation and reports those. Still one compilation per process; memory and simulation are unchanged.
Performance change vs mainWorkflow run · main Clef gate: Running benchmarks because the decision was unavailable (ValueError). The main and PR release binaries are byte-for-byte identical. No compiler performance difference to measure. |
3ca62db to
9a93d97
Compare
Merging this PR will improve performance by 10.02%1 warning
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
|
oops didn’t mean to merge this without approval. let me know if we need to roll back |
codspeed's walltime number came from one compilation per project, so no stdev, no p-value, and a 10% runner swing reads as a change (#669 is docs only and got "improved by 9.7%")
in walltime mode the bench now runs itself ten more times as child processes and reports those. still one compilation per process, memory and simulation modes untouched
local run with the runner env faked, Compiler-Unions: 10 rounds, mean 75.2 ms, stdev 1.4 ms. about 15 s more per project on the macro runner