Fleet-Wide Root-Cause / Failure-Pattern Retrieval Assistant

A retrieval-augmented system that helps a reliability engineer answer the question every failure investigation starts with: "have we seen this before, and how was it fixed?"

Given a new equipment anomaly (structured fields + a free-text engineer note), the system retrieves the most similar historical incident reports from a fleet-wide corpus, checks whether the matches converge on a single failure mode (a possible systemic pattern) or are scattered (no consensus), and drafts a short, cited briefing for the engineer to review, edit, or reject.

This is a general reliability-engineering knowledge-retrieval pattern used across rotating-equipment industries (turbines, compressors, generators) — it is not a reproduction of any specific company's internal tooling.

On the data: the incident corpus is entirely synthetic, generated by an LLM against a structured failure taxonomy (src/taxonomy.py) so the patterns are structurally realistic without describing any real unit, company, or incident. This is disclosed here and should be disclosed in interviews — it's what makes the project defensible.

Architecture

New anomaly (structured fields + free text)
        |
        v
  query_builder.py  -->  retrieval query
        |
        v
  retrieval.py
    dense search (Chroma + OpenAI embeddings)  --\
    sparse search (BM25, exact fault-code match) --> Reciprocal Rank Fusion --> cross-encoder rerank
        |
        v
  clustering.py  -->  majority-vote pattern analysis (systemic vs. isolated)
        |
        v
  generate_briefing.py  -->  cited, human-review-first recommendation (OpenAI chat model)
        |
        v
  app.py (Streamlit)  -->  interactive demo UI

Setup

cd fleet-rca-assistant
python -m venv .venv && source .venv/bin/activate   # or .venv\Scripts\activate on Windows
pip install -r requirements.txt
cp .env.example .env   # then add your OPENAI_API_KEY

Build order (matches the 4-week plan)

Week 1 — Data

python src/generate_data.py --n 400 --out data/incidents.json

Generates ~400 synthetic incident reports. Spot-check a sample of the output for realism/diversity before moving on — this is the foundation everything else sits on.

Week 2 — Retrieval

python src/ingest.py --in data/incidents.json

Builds the Chroma dense index and the BM25 sparse index. Then sanity check retrieval quality directly in a Python shell:

from src.retrieval import hybrid_retrieve
hybrid_retrieve("bearing temperature climbing, metallic smell", top_n=5)

Week 3 — Generation + UI

streamlit run app.py

Walk through the sidebar: pick an equipment class/component, enter an engineer note, click "Find similar cases." This exercises the full pipeline end to end (query building → hybrid retrieval → pattern clustering → briefing generation).

Week 4 — Evaluation

1. Open eval/labeled_queries.json and replace the template with 30-50 real hand-labeled query/expected-failure-mode pairs, using entries from your generated data/incidents.json as ground truth.
2. Run:

   python eval/run_eval.py

3. Extend with a RAGAS faithfulness/answer-relevancy pass on a sample of generated briefings (stub noted in eval/run_eval.py).
4. Write up results in this README, add an architecture screenshot from the Streamlit app, and record a short demo.

Project structure

fleet-rca-assistant/
├── app.py                    Streamlit demo UI
├── requirements.txt
├── .env.example
├── data/
│   └── incidents.json        generated by generate_data.py (not committed if large)
├── src/
│   ├── taxonomy.py           controlled vocabulary (equipment/component/failure-mode)
│   ├── generate_data.py      synthetic incident-report generator
│   ├── ingest.py              builds Chroma + BM25 indexes
│   ├── retrieval.py          hybrid search + RRF fusion + cross-encoder rerank
│   ├── query_builder.py      structured anomaly -> retrieval query
│   ├── clustering.py         systemic-pattern detection
│   └── generate_briefing.py  cited recommendation generation
└── eval/
    ├── labeled_queries.json  hand-labeled eval set (template — fill in)
    └── run_eval.py           precision@k + RAGAS scaffold

Design notes worth calling out in an interview

• Precision@k on labeled failure_mode, not generic similarity — the metric that matters here is whether retrieval finds cases that share the true root cause, not cases that merely sound similar in wording.
• RRF fusion over dense+sparse — dense embeddings catch semantic symptom similarity; BM25 catches exact fault-code/part-number matches that embeddings often blur. Fusing both is more robust than either alone.
• "No good match" is a valid output — the system is designed to say so rather than force a low-confidence guess (see analyze_pattern's confidence score and the briefing prompt's explicit instruction).
• Human-in-the-loop by design — the generated briefing is explicitly framed as a draft for engineer review, not an autonomous decision.

Future enhancements

• Replace majority-vote clustering with embedding-space clustering (HDBSCAN) for fleet-scale pattern detection across thousands of units.
• Add a feedback loop: engineer approve/reject signal on briefings feeds back into reranker fine-tuning.
• Swap the OpenAI embedding/chat calls for a self-hosted open-weight model (Llama 3.1) behind the same interface, and benchmark cost/ latency/quality trade-offs — the retrieval and clustering code doesn't need to change, only generate_briefing.py and ingest.py's embedding function.