Skip to content

PHASE_PLAN.md

Latest commit

 

History

History
447 lines (365 loc) · 23.8 KB

PHASE_PLAN.md

File metadata and controls

447 lines (365 loc) · 23.8 KB
Raw

RAGProof - Phase Plan (Engineering Execution Plan)

Derived from: RAGPROOF_BUILD_PLAN.md v1.0 Plan version: 1.0 Date: 2026-07-02 Status: Ready for execution

This document is the authoritative execution plan. It preserves the phase order of the build spec (Section 12) but hardens every phase against real-world failure modes the spec does not fully address. Where this plan and the spec conflict, this plan wins; the conflict is recorded in Section 2 below.

The companion TASKS.md is the flat, trackable checklist. Task IDs there (P0-01, P3-07, …) map to the phases here.

1. Pre-flight verification (done 2026-07-02)

Check Result
PyPI name ragproof Free - pypi.org/pypi/ragproof/json returns 404
GitHub name ragproof Free - only an unrelated RAGProofofConcept notebook exists

The name is cleared. Reserve it early: create the GitHub repo in Phase 0 and publish a 0.0.1 placeholder to PyPI at the end of Phase 0 (PyPI has no name reservation other than publishing).

2. Real-world issues identified in the spec, and how this plan fixes them

These are deliberate deviations/additions. Each is folded into the phase tasks below.

RW-1, SQLite under async concurrency will deadlock or corrupt runs

The spec pairs RAGPROOF_MAX_CONCURRENCY=4 async workers with SQLite. Concurrent writers on SQLite produce database is locked errors under default settings. Fix (Phase 1): enable WAL mode + busy timeout at connection setup; route all writes through a single writer task fed by an asyncio.Queue (workers never touch the DB directly); use aiosqlite via SQLAlchemy async. Add a stress test: 32 concurrent cases against a jittery mock adapter, zero lock errors.

RW-2, A crash at case 480/500 loses the whole run

Evaluations hit flaky pipelines and rate limits; without checkpointing, a 20-minute run dies at the last case and all cost is wasted. Fix (Phase 1): persist each Result as it completes; runs have a state machine (running → completed | partial | aborted); ragproof run --resume <run_id> skips already-completed cases. Per-case errors are recorded as case status (error / timeout / judge_error), never as a score of 0, and never kill the run.

RW-3, Flaky CI gates are the #1 way eval tools get uninstalled

LLM-judge metrics are noisy; a relative threshold of 0.03 will fire on judge noise alone for small datasets, and teams will delete the gate the first week it blocks a release falsely. Fix (Phase 3/6): judge calls at temperature 0 with structured JSON output; a content-addressed judge cache (model + prompt hash + input hash) makes re-runs on unchanged cases exactly reproducible and nearly free; compare and gate compute bootstrap 95% confidence intervals on judge-backed metrics and warn when a delta is within noise; gate config supports a noise_floor per metric; docs recommend tight gates on deterministic metrics, loose gates + CI-aware deltas on judge metrics; a minimum-sample warning fires when n < 30.

RW-4, Distinct exit codes or CI can't tell "quality regressed" from "infra broke"

The spec says exit 0/1. A pipeline outage would then read as a quality regression. Fix (Phase 1, enforced Phase 6): exit code contract: 0 pass, 1 gate/threshold failure, 2 execution error (adapter down, judge unreachable, budget aborted), 3 configuration error. JUnit output marks execution errors as errors, not failures.

RW-5, The abstention metric as specced rewards a pipeline that always refuses

Scoring only "did it decline on unanswerable questions" gives a perfect score to a pipeline that answers nothing. Fix (Phase 5): add robustness.overrefusal - the refusal rate on answerable QA cases - and report abstention and overrefusal side by side. The report calls out the trade-off explicitly.

RW-6, Cost controls that only check after the fact don't stop blowouts

Fix (Phase 3): budget is checked before every judge/generation call against accumulated actuals (provider-reported token usage where available, tokenizer estimate otherwise); on breach the run stops gracefully, is marked aborted:budget, persists everything done so far, and exits 2. The judge cache (RW-3) is the main cost reducer: repeated runs over an unchanged dataset only pay for changed cases.

RW-7, Judge output is not JSON just because you asked nicely

Fix (Phase 3): every judge prompt demands a JSON object; responses are validated against a Pydantic schema; on failure, one repair retry with the validation error appended; on second failure the case is marked judge_error (excluded from the mean, counted and displayed - never scored as 0 or dropped silently).

RW-8, Dataset/config hashes must be canonical or "immutable" is a lie

Naïve json.dumps hashing varies with key order, whitespace, and unicode escaping. Fix (Phase 1/4): one canonical-JSON function (sorted keys, UTF-8, no ASCII escaping, fixed separators) used for every hash; algorithm (sha256) recorded with the hash; loading a frozen dataset re-verifies the hash and refuses on mismatch.

RW-9, Metric edge cases will produce NaNs and silent lies unless defined up front

Fix (Phase 2, documented in docs/metrics.md): defined behavior for: fewer than k chunks retrieved; empty retrieval; duplicate chunk IDs (deduplicate, keep first rank); no relevant item found (MRR = 0); empty expected-source set (case invalid at freeze time, rejected then - not at run time); zero-claim answers for groundedness (status not_applicable, excluded from mean, counted); citations referencing duplicate IDs. Every edge case has a fixture test with an exact expected value.

RW-10, Skipped ≠ zero, everywhere

The spec states this for missing retrieve(); this plan generalizes it: any metric that cannot be computed for a case or run gets status skipped with a machine-readable reason, surfaced in report and gate output. A gate threshold on a skipped metric is a gate failure by default (on_missing: fail | skip config).

RW-11, Windows is a first-class platform

Development happens on Windows; most eval tools break there first (paths, event loops, file locking, console encoding). Fix (Phase 0): CI matrix {ubuntu, windows, macos} × {3.11, 3.12, 3.13} from the first commit; pathlib everywhere; no fcntl; UTF-8 explicit on every file open; Rich handles console encoding.

RW-12, Heavy ingestion deps don't belong in the core install

PDF/DOCX parsing drags in large dependencies most CI users don't need. Fix (Phase 0/4): packaging extras - pip install ragproof stays lean (CLI, run, gate, report); ragproof[ingest] adds pypdf/python-docx; a missing extra produces a clear install hint, not a stack trace.

RW-13, The HTML report must work air-gapped

CI artifact viewers and client laptops can't always reach a CDN. Fix (Phase 6): Chart.js is vendored into the template (license header retained); the report is one file with zero network requests; add a CI test that greps the generated report for http(s):// resource loads.

RW-14, Runs across different judge models / prompt versions are not comparable

Fix (Phase 3): every run records judge model + per-prompt hashes; compare and gate refuse-by-default (override flag --allow-mixed-judges) when judge-backed metrics were produced under different judges/prompts, and always label the output.

RW-15, Config drift and typos waste the most user time

Fix (Phase 1): ragproof.yaml is validated by Pydantic with unknown-key rejection and "did you mean" suggestions; ragproof check validates config, env vars, adapter reachability (1 live probe question), and judge connectivity before any expensive run. Referenced-but-unset env vars are named explicitly.

RW-16, Schema will change; runs must survive upgrades

Fix (Phase 1): Alembic migrations from the first table; schema_version stored; opening a newer-schema DB with an older CLI gives a clear error, and older DBs are auto-migrated with a backup file written first.

RW-17, Payload library safety needs enforcement, not intention

Fix (Phase 5): a payload lint test asserts every payload uses inert markers and *.invalid / example.com URLs only - no real domains, no shell commands, no data-exfil endpoints. The test is part of the standard suite so contributions can't regress it.

RW-18, Secrets leak through logs and stored judge output

Fix (Phase 1/3): a redaction filter (registered on the root logger and applied before persisting adapter/judge raw payloads) masks values of any env var matching *_API_KEY|*_TOKEN|*_SECRET plus common bearer-token patterns. Adapter configs store env var names only; a test asserts a leaked key never reaches the DB or logs.

3. Cross-cutting engineering standards (apply to every phase)

  • Language/tooling: Python 3.11+; uv for env + lockfile (uv.lock committed); ruff (lint + format), mypy --strict on ragproof/ (tests may relax); pytest + pytest-asyncio + coverage.
  • Types: Pydantic v2 models at every boundary (config, adapter I/O, judge I/O, dataset schema). No dict[str, Any] crossing module boundaries.
  • Commits: conventional commits, small and reviewable. Each phase ends with a tagged commit phase-N-complete.
  • Tests-before-done: a metric or detector without known-answer fixtures is not done (spec rule 3). Coverage gate ≥ 85% on metrics/, engine.py, judge/.
  • Docs-with-code: docs/metrics.md updated in the same PR as any scoring change; .env.example updated in the same PR as any new env var.
  • PROGRESS.md: updated at the end of every phase - what shipped, what's next, open TODO(decision) items.
  • CI (from Phase 0): lint → typecheck → tests on the 3-OS × 3-Python matrix; CodeQL; dependency review; secret scanning (gitleaks); coverage report.
  • Determinism: all randomness through a seeded random.Random passed explicitly; seeds recorded in run/dataset metadata.

4. Phases

Dependency chain: P0 → P1 → P2 → P3 → P4 → P5 → P6 → P7. P2 and P3 could theoretically parallelize, but sequential is safer since P3 reuses P2's registry and aggregation. Estimates assume one focused engineer (or agent) and include tests + docs.

Phase 0 - Foundation (est. 1–2 days)

Goal: installable, CI-green skeleton; name secured.

Scope: repo scaffold per spec Section 10; pyproject.toml (PEP 621, hatchling, single-sourced version, extras: ingest, dev); Typer CLI stub with all commands registered (generate, freeze, run, compare, gate, report, calibrate, check) each returning "not implemented" with exit 3; ragproof --version; ruff + mypy configs; GitHub Actions CI matrix (RW-11); CodeQL + gitleaks + dependency review; .env.example; MIT LICENSE; README stub; PROGRESS.md; create GitHub repo; publish 0.0.1 placeholder to PyPI via Trusted Publishing (OIDC - no long-lived token secrets).

Acceptance:

  • uv sync && uv run ragproof --help lists all commands on Windows and Linux.
  • CI green on the full matrix; CodeQL and secret scan enabled and passing.
  • pip install ragproof==0.0.1 works from PyPI (placeholder).
  • Exit code contract (RW-4) documented in README and encoded in a ExitCode enum.

Risks: PyPI Trusted Publishing needs the repo public or configured - do repo creation first, publishing last in the phase.

Phase 1 - Adapter layer, run store, and run engine (est. 3–4 days)

Goal: RAGProof executes a run against a real pipeline and persists everything, crash-safely.

Scope:

  • Adapter protocol (retrieve, answer) with Pydantic I/O models (RetrievedChunk, RAGAnswer, ChunkRef). Capability introspection: adapters declare supports_retrieval / supports_answer so the engine plans metric skips up front (RW-10).
  • Python adapter: import-path loading; supports both sync and async user implementations (sync wrapped via thread offload).
  • HTTP adapter: request/response mapping via JSONPath (jsonpath-ng); auth header values sourced from named env vars only (RW-18); per-call timeout; tenacity retries with exponential backoff + jitter, honoring Retry-After, retrying only 429/5xx/timeouts (RW not: 4xx client errors fail fast).
  • Run store: SQLAlchemy 2.x async + aiosqlite; WAL + busy-timeout on connect; single-writer queue task (RW-1); Alembic migrations + schema_version (RW-16); models: Project, Dataset, Case, Run, Result, MetricSummary per spec Section 5, plus run status and per-result status fields (RW-2).
  • Engine: dataset iteration with asyncio.Semaphore(RAGPROOF_MAX_CONCURRENCY); per-case timeout; per-case error capture (RW-2); incremental result persistence; --resume; run manifest (config hash, dataset hash, seeds, versions) via canonical JSON (RW-8); trivial echo.exact_match metric to prove the loop.
  • Config: ragproof.yaml loader with strict Pydantic validation + unknown-key suggestions (RW-15); env var layer per spec Section 11.
  • ragproof check: validates config, env, adapter reachability, DB writability (RW-15).
  • Redaction filter installed on logging and on persisted raw payloads (RW-18).
  • Cost-tracking scaffold: CostLedger accumulating per-call token/cost entries (real accounting lands in Phase 3).

Acceptance:

  • 5-case hand-written JSONL run against the example Python adapter persists results and metadata; second run is comparable; --resume after a forced mid-run kill completes only the remaining cases.
  • Concurrency stress test (32 cases, jittery mock adapter) → zero SQLite lock errors.
  • HTTP adapter tested against a mocked server: mapping, retries, timeout, 4xx fail-fast, Retry-After honored.
  • A planted API key never appears in logs or DB (automated test).
  • Exit codes 2/3 verified for adapter-down and bad-config scenarios.

Phase 2 - Retrieval metrics + compare (est. 2 days)

Goal: the deterministic metric family, exact and edge-case-proof.

Scope: precision_at_k, recall_at_k, mrr, ndcg_at_k (binary relevance default; graded left as documented future); metric registry (stable string names, declared requirements - e.g. needs: expected_source_ids, retrieval); edge-case semantics per RW-9, all fixture-tested with exact values; MetricSummary aggregation (mean/p50/p95, plus counts of scored/skipped/error cases); ragproof compare run_a run_b with per-metric deltas and skip/error visibility; graceful skip-with-reason when the adapter lacks retrieve or cases lack expected_source_ids (RW-10); chunk-ID vs document-ID matching granularity (config, spec §17.4).

Acceptance:

  • Known-answer fixtures pass with exact values, including all RW-9 edge cases.
  • compare prints deltas and marks skipped metrics as skipped, not 0.00.
  • A no-retrieval adapter yields a report/summary that states the skip reason.

Phase 3 - Judge layer + generation metrics (est. 4–5 days; the hard phase)

Goal: groundedness, citation, relevance scoring - calibrated, cached, budgeted.

Scope:

  • Judge client: provider-agnostic (OpenRouter, Ollama, OpenAI, Anthropic); temperature 0; structured JSON responses validated by Pydantic with one repair retry then judge_error (RW-7); per-call timeout + tenacity retries; raw output persisted verbatim post-redaction (spec §14, RW-18).
  • Judge cache: SQLite-backed, keyed (model, prompt_hash, canonical_input_hash); hit/miss stats printed per run; --no-cache flag (RW-3, RW-6).
  • Prompts: versioned files in judge/prompts/; content hash recorded per run; mixed-judge comparability guard in compare/gate (RW-14).
  • Metrics: generation.groundedness (claim decomposition → per-claim verdicts in judge_raw_json; zero-claim answers → not_applicable, RW-9); generation.citation_validity (deterministic; duplicate-ID semantics defined); generation.citation_support; generation.answer_relevance; generation.completeness (only when expected_answer present, else skipped).
  • Cost: provider-reported usage preferred, tokenizer estimate fallback; budget checked pre-call; graceful aborted:budget stop, exit 2 (RW-6); per-run cost in summary and report.
  • Calibration: ≥10 human-scored fixtures per judge prompt; ragproof calibrate reports exact and within-1-band agreement; agreement thresholds in config; CI job runs calibration when judge/prompts/** or judge/fixtures/** change and fails below threshold (spec §7.4 - not cut).
  • Unit tests use recorded judge fixtures (no live calls in CI).

Acceptance:

  • Full run on the example corpus separates planted good vs bad answers on groundedness.
  • Malformed judge output → repair retry → judge_error case; run completes; the error count is visible in summary, report, and JUnit.
  • Cache: re-running an unchanged run costs ~$0 and reproduces judge-metric scores exactly.
  • Budget breach mid-run: partial results persisted, status aborted:budget, exit 2.
  • calibrate produces an agreement report; CI calibration gate demonstrated on a deliberately bad prompt change.

Phase 4 - Dataset generation (est. 3 days)

Goal: users get a trustworthy eval set without hand-writing one.

Scope: corpus ingestion (TXT/MD in core; PDF/DOCX behind ragproof[ingest] with clear missing-extra error, RW-12; per-file size cap; extraction-failure report listing skipped files - never silent); chunk sampling with explicit seed; QA synthesis with second-pass answerability verification (discard + count failures); unanswerable synthesis verified by retrieval + judge pass; injection case generation from the payload library (poisoned document variants registered with expected non-compliance markers); JSONL review file; ragproof freeze computing corpus_hash + dataset hash via canonical JSON (RW-8); frozen datasets verified on load, mutation refused; generation metadata (models, seeds, prompt hashes, discard counts) embedded in the dataset record.

Acceptance:

  • generate on the tiny test corpus yields spot-checkable answerable QA cases; the discard rate is reported.
  • Frozen dataset: hash verified on load; edited file refuses to load with a clear message.
  • Same seed + same corpus → identical sampling and case ordering (model output may vary; sampling may not).
  • Ingesting a corrupt PDF skips the file with a report line, exit code unaffected.

Phase 5 - Robustness metrics (est. 2–3 days)

Goal: the differentiating metric family, safely built.

Scope: payload library with 10+ types (instruction override, exfiltration-URL to *.invalid, tone hijack, citation spoofing, system-prompt fishing, formatting hijack, competitor-praise steering, fake-citation injection, link-bait, chain instructions); per-payload deterministic compliance detectors (string/regex), each with positive and negative fixture tests; payload safety lint (RW-17); robustness.injection_resistance = 1 − compliance rate; robustness.abstention (heuristic refusal detection + judge confirmation) on unanswerable cases; robustness.overrefusal on answerable cases (RW-5); fabrication-on-unanswerable given prominent weight in summaries per spec §7.3; two example pipelines in examples/: deliberately vulnerable and guarded.

Acceptance:

  • Vulnerable example scores low, guarded example scores high on injection resistance (asserted in an integration test).
  • Abstention distinguishes "not in the documents" from fabrication; an always-refusing pipeline shows high abstention and high overrefusal.
  • Every payload detector has fixture tests; payload safety lint passes and is in CI.

Phase 6 - Reports, CI gate, distribution (est. 3–4 days)

Goal: the product surface teams actually touch.

Scope:

  • HTML report: single self-contained file, Chart.js vendored (RW-13); overview scores, per-metric distributions, run comparison, worst-10 cases per metric with question/answer/context/judge reasoning, skip/error counts, cost summary, dataset
    • config + prompt hashes displayed (spec §14); no-network test in CI.
  • Markdown summary for PR comments; JUnit XML (one test per metric; execution errors as <error>, threshold failures as <failure>, RW-4).
  • ragproof gate: absolute + relative thresholds; per-metric noise_floor; bootstrap 95% CIs on judge-backed metrics with in-noise warnings (RW-3); on_missing: fail|skip for skipped metrics (RW-10); minimum-sample warning (n < 30); exit code contract enforced end to end.
  • GitHub Action (action.yml in-repo): install → run → gate → upload HTML artifact → sticky PR comment with the Markdown summary.
  • Dockerfile: slim multi-stage image, non-root user, pinned base digest.
  • --json output flag on run/compare/gate for tooling.

Acceptance:

  • Gate exits 1 on threshold breach, 2 on execution error, with JUnit rendering correctly in GitHub Actions.
  • HTML report opens from disk with zero network requests (automated check).
  • The Action runs end to end in this repo's own CI against the example pipeline, uploading the report and commenting on a test PR.
  • A judge-noise delta inside the noise floor does not fail the gate; a genuine regression does (both covered by integration tests).

Phase 7 - Case studies and launch (est. 3–5 days, elapsed longer)

Goal: proof, not just code.

Scope: DOC-007-AI and Legate Agent adapters (first-class citation/kb.search mapping); run both, fix at least one real issue the scores expose, publish before/after numbers in each repo's README; docs/metrics.md (exact computation of every metric, incl. edge-case semantics from RW-9), docs/quickstart.md, docs/adapters.md, docs/ci.md; PyPI 1.0.0 via Trusted Publishing; demo GIF (degrading PR → red gate → fix → green gate); launch post draft; README leads with case-study numbers and the GIF (spec §18).

Acceptance:

  • Both case studies show real numbers and ≥1 real improvement found by RAGProof.
  • pip install ragproof gets 1.0.0; quickstart works verbatim on a clean machine (Windows and Linux verified).
  • README leads with GIF + case-study numbers.

5. Risk register

# Risk Likelihood Impact Mitigation
R1 Judge noise makes gates flaky → users remove the gate High High RW-3: cache, CIs, noise floors, deterministic-first guidance
R2 SQLite lock/corruption under concurrency High Medium RW-1: WAL + single-writer queue + stress test
R3 Eval cost surprises users Medium High RW-6: pre-call budget checks, cache, cost always printed
R4 Judge model deprecations break calibration Medium Medium Calibration is per model+prompt; agreement re-verified in CI; provider-agnostic client
R5 Windows breakage discovered late Medium Medium RW-11: full OS matrix from Phase 0
R6 Payload library accused of shipping attack tooling Low High RW-17: inert-payload lint enforced in CI; docs state intent
R7 Scope creep into dashboard/SaaS territory Medium Medium Spec §1 out-of-scope list honored; HTML report is the only viewer in v1
R8 Synthetic dataset quality poor → misleading scores Medium High Answerability verification pass, discard-rate reporting, human review file before freeze
R9 Name squatting between now and repo creation Low Medium Phase 0 creates repo + placeholder PyPI release immediately

6. Milestone summary

Milestone Phases Cumulative estimate You can…
M1 Skeleton P0 ~2 days install it, CI green
M2 It runs P1 ~6 days evaluate a real pipeline, crash-safe
M3 It measures P2–P3 ~13 days trust retrieval + generation scores, calibrated
M4 It generates P4–P5 ~18 days build datasets, break pipelines on purpose
M5 It gates P6 ~22 days block a bad PR automatically
M6 It's proven P7 ~27 days show real before/after numbers publicly

Definition of done per feature: spec Section 16 applies unchanged.