feat(helm): add Prometheus operator ServiceMonitor support

[aWN4Y25pa2EK]

Problem

The api (port 8888) and worker (port 8889) containers expose Prometheus format metrics at /metrics, but the chart wires scraping for neither: the worker Service's scrape annotations are gated behind the unrelated podAnnotations value (dead by default), the api Service has no annotations at all, and no ServiceMonitor exists. The dev LGTM stack also only scraped the api — worker metrics were not collected in any shipped configuration.

What changed

• templates/servicemonitor.yaml (new): per-component ServiceMonitor resources — api whenever api.enabled, worker only when worker.enabled — gated on metrics.serviceMonitor.enabled (default false).
• values.yaml: metrics.serviceMonitor block (labels for operator selection, path, interval, scrapeTimeout) with operator prerequisites documented.
• README.md: values table rows + "Prometheus metrics" section.
• scripts/dev/monitoring/prometheus.yml: add hindsight-worker job (host.docker.internal:8889).

The worker monitor targets the headless Service, so each StatefulSet replica gets its own scrape target (hindsight-worker-0/1). The control plane exposes no metrics (Next.js, no instrumentation) and is intentionally excluded.

Usage

metrics:
  serviceMonitor:
    enabled: true
    labels:
      release: kube-prometheus-stack  # match the stack's serviceMonitorSelector

Validation

• helm lint clean; renders verified: 0 monitors by default, 1 with api-only, 2 with worker.enabled=true; selector labels stable (name/instance/component); user labels land on monitor metadata.
• End-to-end on k3d + kube-prometheus-stack: installed chart with worker.enabled=true, metrics.serviceMonitor.enabled=true, metrics.serviceMonitor.labels.release=kps; both monitors created and reconciled; Prometheus active targets:
  • hindsight-api http://10.42.0.17:8888/metrics health=up
  • hindsight-worker http://10.42.0.18:8889/metrics health=up
  • hindsight-worker http://10.42.0.19:8889/metrics health=up
  • up == 1 for all three jobs; api served 113 hindsight_* samples, worker 28 (db pool, process, poller families).

Notes

• Requires the monitoring.coreos.com/v1 CRDs; without the operator nothing is emitted (default off), so plain installs are unaffected.
• Unrelated chart findings from a full assessment (missing values.schema.json, broken existingSecret + external-DB path, duplicate env entries under SSA, pgvector:latest pin) are tracked separately and not addressed here.

[strix-security]

Strix Security Review

No security issues found.

_{Updated for 64dee42.}

---

Reviewed by Strix
_{Re-run review · Configure security review settings}