IntelligenceOS Risk Evaluation Harness

Leakage-safe, reproducible evaluation of imbalanced financial-risk classifiers.

Evidence status: Benchmarked Research Prototype Domain: Financial Risk · Fraud Detection · Model Evaluation

This repository compares three classical model strategies on the public ULB/Kaggle credit-card fraud dataset. It is an engineering and benchmark artifact—not a deployed fraud service, client result, or production-performance claim.

Verified benchmark

The execution pass independently reran every model with Python 3.11.15 and the pinned dependencies in pyproject.toml. Unlike the earlier result, the corrected workflow selects each F1 threshold on validation labels and evaluates the chosen operating point once on a held-out test split.

Model	Average Precision	ROC AUC	Precision	Recall	F1	FP	FN	Validation-selected threshold
Logistic Regression	0.7192	0.9720	0.5526	0.8571	0.6720	68	14	0.99
SMOTE + Random Forest	0.7920	0.9768	0.7500	0.8265	0.7864	27	17	0.71
XGBoost	0.8756	0.9749	0.9011	0.8367	0.8677	9	16	0.54

Evidence label: public-dataset benchmark. The machine-readable evidence is results/benchmark.json, including the dataset SHA-256, split details, configuration, environment, model parameters, thresholds, and full metrics.

The prior 0.8693 XGBoost result used final test labels to select its reporting threshold. Average precision itself is threshold-independent, but the old threshold-dependent precision/recall/F1/FP/FN figures were not defensible held-out estimates and have been superseded.

Leakage controls

creditcard.csv
    ↓ validate schema, labels, numeric/finite values, and fingerprint
stratified train / validation / test split (60% / 20% / 20%)
    ├── train: fit Amount scaler; fit model; apply SMOTE only here for RF
    ├── validation: select operating threshold by F1
    └── test: score the selected threshold once
    ↓
JSON evidence artifact + console comparison

These boundaries are independently testable. Changing final test labels cannot change the selected threshold, scaler metadata records train_only, and SMOTE provenance records its train-only scope.

Quick start

python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install ".[dev]"
python -m pip check
ruff check .
ruff format --check .
pytest -q

Download the public dataset and place it at data/raw/creditcard.csv:

python -m pip install kaggle
kaggle datasets download -d mlg-ulb/creditcardfraud -p data/raw --unzip

The verified file has 284,807 rows, 31 columns, and SHA-256:

76274b691b16a6c49d3f159c883398e03ccd6d1ee12d9d8ee38f4b4b98551a89

Run all three models:

ios-eval \
  --data data/raw/creditcard.csv \
  --config configs/benchmark.json \
  --output results/benchmark.json

The macOS XGBoost wheel requires the OpenMP runtime (brew install libomp). Linux wheels used by CI and the Docker image include their supported runtime path.

Reproducible checks

docker build -t ios-risk-eval:test .
docker run --rm ios-risk-eval:test

GitHub Actions runs Python 3.11 dependency validation, Ruff lint/format checks, 32 automated tests, and the same test suite inside Docker. The public dataset is intentionally not required for CI and is never committed.

Repository structure

src/ios_eval/
├── data_loader.py   # validation, deterministic splits, train-fitted scaling
├── metrics.py       # metrics, threshold sweeps, deterministic selection
├── evaluation.py    # model fitting, train-only SMOTE, provenance orchestration
├── report.py        # atomic JSON serialization and console comparison
└── cli.py           # ios-eval command
configs/benchmark.json
tests/
results/benchmark.json
docs/
pyproject.toml
Dockerfile

The historical nested project wrapper, empty modules/notebooks/config, duplicate leaky scripts, oversized environment dump, stale single-model result, and obsolete marketing drafts were removed.

Evidence boundaries and limitations

• One historical public dataset does not establish performance on another processor, geography, product, fraud regime, or live client stream.
• The selected threshold optimizes validation F1, not a real institution's fraud-loss/review-cost function.
• This is a single deterministic split, not temporal or out-of-time validation.
• No calibration, drift monitoring, serving latency, SLA, production deployment, or client outcome is claimed.
• Average precision measures ranking quality; threshold-dependent metrics describe only this selected operating point and held-out split.

See architecture, evidence policy, failure modes, and benchmark reproduction.

License

Copyright 2026 Ugo Chukwu / Etherlabs. Licensed under the Apache License, Version 2.0.