DRAFT: PoC OIE orchestrator (Modules A→B→C)

[northdpole]

Summary

• Adds a PoC/demo orchestrator that sequences GSoC Module A (Harvester) → Module B (Noise Filter) → Module C (Librarian) for one pipeline_run_id.
• Does not modify Modules A/B/C. Missing entry points become TODO: / todo stage results in the orchestrator only.
• CLI: python scripts/run_oie_demo_pipeline.py (Makefile: make oie-demo-pipeline).

GSoC A/B/C readiness (PR survey, 2026-07-24)

Module A — Harvester (stacked #920 → #983 → #985 → #986 → #987)

Independently runnable?	Partially — library components + unit tests only
PoC/Demo ready?	No end-to-end. Tip #987 has config/git/change-detect/file-filter/diff retrieve·parse·normalize, but no chunking/ChangeRecord emit, no harvest_input writer, no CLI. #987 CI Test failing.
Orchestrator entry?	No — no run_harvester / --run_harvester

Module B — Noise Filter (merged W1–4 #913/#928/#947/#976; open #989)

Independently runnable?	Yes — stages on main via scripts/evaluate_noise_filter.py (needs LLM for Stage 2)
PoC/Demo ready?	Stages yes; full DB pipeline needs #989 (harvest_input/knowledge_queue + run_noise_filter)
Orchestrator entry?	Yes on #989: noise_filter.pipeline.run_noise_filter(session, pipeline_run_id, …) and cre.py --run_noise_filter --run_id …

Module C — Librarian (merged W1–4; open #974 → #990 → #991)

Independently runnable?	Yes dry-run on main: python cre.py --librarian_dry_run [--librarian_source JSONL] (C.0→C.2)
PoC/Demo ready?	Yes for fixture JSONL dry-run (needs embeddings DB / API). No graph writes (W8). Full C.0→C.4 envelopes on #991 (LibrarianPipeline) still dry-run.
Orchestrator entry?	Yes (partial) — cre_main.run_librarian on main; LibrarianPipeline.run(at=…) on #991 (not yet wired into cre_main)

Orchestrator behavior on this PR

1. A — default skipped; --run-a records todo (no entry to call).
2. B — calls run_noise_filter if importable (Add harvest_input and knowledge_queue models with pipeline support #989); else todo.
3. C — calls run_librarian (works on main today with fixture JSONL). Notes when LibrarianPipeline is importable (week_6b: Module C (The Librarian) — C.4 envelope emitter + C.0→C.4 pipeline glue #991+) but still uses run_librarian until a component factory exists.

Test plan

• python -m unittest application/tests/oie_orchestrator_test.py
• python scripts/run_oie_demo_pipeline.py --skip-b --skip-c smoke
• After Add harvest_input and knowledge_queue models with pipeline support #989 merges: seed harvest_input, run with --skip-a (no --skip-b)
• Librarian-only demo: --skip-a --skip-b --librarian-source application/tests/librarian/fixtures/sample_knowledge_queue.jsonl against a DB with CRE embeddings
• Re-check TODOs after Module A lands run_harvester and after week_6b: Module C (The Librarian) — C.4 envelope emitter + C.0→C.4 pipeline glue #991 wires LibrarianPipeline into CLI/DB source

Depends on (not included)

• Module A entry point (post-GSoC Module A: week 5 : feat(harvester): add git diff retrieval pipeline (stacked on top of #986) #987)
• Add harvest_input and knowledge_queue models with pipeline support #989 Module B pipeline tables/CLI
• week_6b: Module C (The Librarian) — C.4 envelope emitter + C.0→C.4 pipeline glue #991 Module C LibrarianPipeline + eventual W8 DB source / graph write

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro Plus

Run ID: 60ad7e9d-6717-4807-b122-dc090aed3edb

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/oie-poc-orchestrator

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}