feat(drift): evaluation metrics for drift detectors

[jevwithwind]

Hi, as discussed in #1914, this is a prototype evaluation tool for drift detection at 'river/drift/evaluate.py'. It's only a draft and needs help on the API side.

I used claude code to write the code and tests, while I directed it and reviewed its output before putting it here.

It's my first time contributing here and please let me know what you make of this.

[codspeed-hq]

Merging this PR will not alter performance

✅ 96 untouched benchmarks
⏩ 16 skipped benchmarks¹

---

_{Comparing jevwithwind:drift-evaluate (bdc845a) with main (df2466f)}

Footnotes

1. 16 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[jevwithwind]

For metrics definitions, I used:

1. Cerqueira et al. 2026 (preprint) since two of the authors are on the River team and it implements the protocol in CapyMOA, so claude matched it.
2. Bifet 2017 (doi 10.1007/978-3-319-59060-8_64), which is where the four metrics in Drift module roadmap: broaden detector coverage + build an evaluation harness (6-month focus) #1914 come from.
3. Lukats et al. (doi 10.1007/s41060-024-00620-y), recommended by @e10e3.

[jevwithwind]

To continue, I want to know: does this belong in river/drift/ or river/evaluate/?
And should delta_max should keep having no default following Cerqueira et al. 2026?
Also any clue on how to proceed with the API side.

[e10e3]

Thank you for the contribution!

If all additions here are metrics, maybe they should go to the metrics module, or perhaps one of its submodule?
If you chose this option, I shall note that all metrics (and much of River's API) are implemented as classes, whereas yours are functions. To integrate them in the metrics, you will need to turn them into classes.

metrics already has metrics with the same name as yours (F1, Precision, Recall…), is there no way to reuse them?

The article from Cerqueira et al. that you mention says that $\delta_\mathrm{pre}$ should be 0 only for synthetic streams because of how they're constructed, so 0 is not a good value for all possible streams. If you need to have defaults for $\delta_\mathrm{pre}$ and $\delta_\mathrm{max}$, you could consider what other papers use. Bifet's paper appears to use $\delta_\mathrm{pre} = 0$ and $\delta_\mathrm{max} = +\infty$ (this shows that both papers have differing evaluation methodologies).

You implementation relies on inputing Iterables of results. Does this work well with River's iterative behaviour? We don't want to need to store all of a model's predictions to evaluate it. The current metrics work the following may:

for x, y in dataset:
    y_pred = model.predict_one(x)
    metric.update(y, y_pred)
   model.learn_one(x, y)

Where, at every step, we update the metric with the model's output, so we only need to store the aggregate and not the full output series of the model. This also allows us to get a reading of the metric at any time without additional computation.
This way of working is central to River, and we want to preserve it.

[jevwithwind]

Thank you for the reply! Yes, metrics should be in classes not functions, and putting them in metrics makes sense. δpre default to 0 should be removed too.

Regarding the Iterables, understood, the code is fixable to fit the existing frame. The drift schedule is known in advance so each step can be classified as it arrives.

Regarding the existing metrics, F1, precision, recall can be reused but episode recall counts windows not alarms. NDT and the rest are not classification metrics. To reuse them, I would add a matching step (alarm stream -> drift schedule -> TP/FP/FN under a tolerance window) before reusing the existing arithmetic.

Claude also made a mistake with Bifet's paper on how it counts drifts. I will start with the rework shortly.