scriptengine-bench

A/B performance benchmark for ScriptEngine on HPC (Lustre).

Compares two SE branches by running your existing EC-Earth4 experiment setup+config phase N times and reporting the median wall-clock time.

Quick start

# Clone into your HPC scratch
cd /gpfs/scratch/.../your-workspace
git clone https://github.com/vlap/scriptengine-bench.git
cd scriptengine-bench

# Run the benchmark from your experiment's runtime directory
cd /path/to/your-experiment/runtime
bash /path/to/scriptengine-bench/bench.sh <branch-to-test> [N]

Usage

bench.sh <branch> [iterations] [se-repo-url]

  branch       Branch/tag/commit to test against master
  iterations   Number of A/B iterations (default: 3)
  se-repo-url  ScriptEngine git URL (default: https://github.com/uwefladrich/scriptengine.git)

The script will:

1. Clone SE (or use an existing clone in ~/.cache/scriptengine-bench/)
2. Build two source trees: master and <branch>
3. Auto-detect your experiment YAML files from the current directory
4. Ask you to confirm the run directory (the dir created by setup)
5. Warm up Lustre page cache
6. Alternate timed runs of both versions
7. Report median timings and percentage difference

Example

cd /gpfs/scratch/cns82/bsc032446/aav1/ecearth4/runtime
bash ~/scriptengine-bench/bench.sh perf/parallel-loops 5

Output:

=== SUMMARY (median of 5 runs) ===
  perf/para  setup:    7.6s    config:   4.5s
  master     setup:   17.7s    config:   8.7s

  setup  diff: 10.1s (57% faster)
  config diff: 4.2s (48% faster)

Requirements

• Bash 4+
• se in PATH (conda env with ScriptEngine installed)
• An existing EC-Earth4 experiment (already configured, with inidata accessible)
• Git (to clone/checkout SE branches)