Add train/test splits so hillclimbing can detect overfitting

[mattpodwysocki]

Stacked on #87 — base is eval-runner-improvements. Retarget to main once that merges.

Why

Every eval was effectively a training eval. Iterating on a skill to raise its scores had no way to tell "the skill got better" from "the wording now matches the expectations". That's the leak eval-design guidance warns about, and the suite had no defence against it.

The split

Evals marked "split": "test" are held out; unmarked ones are train. 27 of 89 held out (~30%), assigned as every third eval per skill by id — so each skill contributes to both sets, and the assignment is reviewable rather than trusted to a random seed.

node scripts/eval.js --split=train   # tune against these
node scripts/eval.js --split=test    # read these to see if it generalized
node scripts/eval.js                 # both (default)

The overfitting signal

The summary breaks results out per split, and --diff reports train and test movement separately:

Train vs test:
  train  94.1% → 97.2% (+3.1)
  test   93.8% → 93.9% (+0.1)

⚠️  Train improved while test did not. That is the signature of a change
    fitted to the eval set rather than to the skill — consider reverting.

Documented

CONTRIBUTING.md covers the loop (baseline → edit → diff) plus the two cautions that matter most here:

• Use --repeats before believing a delta. One eval in this repo scored 15/15 and 11/15 on identical content. A change smaller than an eval's own spread cannot be attributed to the change.
• Don't write new evals from model failures. Selecting cases the current model gets wrong fits the suite to one model's blind spots rather than to what users actually get wrong. Hard cases should come from a support escalation, a bug report, or a domain expert calling a case hard.

What this does not fix

This closes the infrastructure side of hillclimbing. With #86 and #87 in place, the suite now has a valid scaling signal (monotonic across three model tiers), variance measurement, errors excluded from scores, and an overfitting check.

The remaining gap is headroom: 67% of skill-surface evals score 100%, so most of the suite can register a regression but not an improvement. That needs harder cases from real sources, which tooling cannot manufacture — and the caching eval added in #80 is the worked example, since it came from an actual escalation and landed at 87% rather than pinning to 100%.

Test plan

• npm run check passes
• --split=train|test|all verified to select 12 / 6 / 18 runs on mapbox-navigation-patterns
• Per-split breakdown verified on a multi-skill run
• --diff against a baseline predating splits degrades gracefully rather than erroring

🤖 Generated with Claude Code