Skip to content

feat(admin): detection quality metrics (reject rate, missing source and proof counts) - #137

Merged
vidit-admin merged 4 commits into
mainfrom
claude/main-update-next-steps-af29ad
Jul 16, 2026
Merged

feat(admin): detection quality metrics (reject rate, missing source and proof counts)#137
vidit-admin merged 4 commits into
mainfrom
claude/main-update-next-steps-af29ad

Conversation

@vidit-admin

@vidit-admin vidit-admin commented Jul 16, 2026

Copy link
Copy Markdown
Member

Two things land here: an admin-only detection-quality feature and the tracker upkeep around it.

Feature: admin detection-quality metrics

A new Detection quality panel in the admin page, admin-only, surfacing the health of the machine-extraction pipeline. Backed by a read-only GET /admin/detection-stats (🛡️).

Reject-rate. The share of machine detections dismissed while still a draft, whichever door they left through. A machine detection is a row imported from X (the archive backfill or the bot), discriminated by detected_from_url being set (a human submit carries null); demo rows (is_demo) are excluded. A detection counts as "rejected" if either an owner closed it straight out of detected (status = "closed" with before_closed_status = "detected") or an admin soft-deleted it while it was still detected (deleted_at set with status = "detected"). This matches the services/detection._reimportable dismissal semantics, where soft-delete and owner close are the same judged-and-thrown-out shape. A detection vouched into geolocated (even one later soft-deleted, it was vouched before removal), or one still awaiting review, is not a reject. reject_rate = machine_rejected / machine_total as a 0..1 ratio (0 when there are no machine detections), counted over every non-demo machine row (soft-deleted or not, since the metric measures what the pipeline produced). Two accepted counting edges, both preferring over-counting dismissals: an owner hard-delete drops the row from both counts, and an account-departure cascade soft-delete counts that account's pending drafts as rejects.

Pending quality counts. Over the live detected queue (deleted_at IS NULL, machine rows only, demo excluded), three cheap signals count drafts missing a piece the geolocate floor will demand: no source media, no proof-role image (a video in the proof slot still lacks an image), no source_url.

Shape. Business logic in services/admin.detection_quality_stats (two grouped count(*) FILTER (WHERE …) aggregates), a Pydantic AdminDetectionStatsRead, admin-only auth like the other admin endpoints. The frontend adds DetectionStatsPanel composed from existing ui primitives (admin surfaces inline their markup, no new shared primitive), with the exact definitions documented in the card and in docs/api.md. Backend tests pin the definitions (a closed-from-detected row is a reject, a soft-deleted draft is too, a geolocated one is not even once soft-deleted, a human submit moves neither total, a demo row moves nothing, and the pending counts flag each missing piece, a proof video counting as a missing image, while excluding soft-deleted rows). api-types.ts regenerated so the drift gate stays green.

Tracker sync (already on the branch)

The earlier commit synced planning/next.md with work shipped on main: the ingest-contract test suite (#136) folded into the Landed paragraph, and the P0 archive on-ramp row deleted (its "Tag @ViditBot" half stays covered by the Bot rows).

Tracker moves

  • Deleted the now-shipped P1 Detections — Admin machine reject-rate metric row.
  • Moved two descoped v0.4 backfill rows to Unscheduled candidates (concept-level, keeping the why): Archive note-tweet.js long-form bodies and Archive authenticity sample-check. Not needed for v0.4; promoted back if a real archive needs them. The v0.4 "What's left" prose is restated to match.

Gates

Backend (backend/): ruff check ., ruff format --check ., mypy app (80), vulture, pytest (831 passed) — all green. Frontend (frontend/): tsc --noEmit, eslint . (0 errors), vitest (179), next build — all green. Repo root make hygiene (jscpd, knip, vulture, palette-coverage) — green. api-types.ts regenerated via make gen-api-types.

🤖 Generated with Claude Code

…-contract suite

The archive-upload on-ramp (POST /events/import-archive, in-browser strip,
/submit bulk mode, confirm-registration landing) and the per-typology
ingest-contract test suite both shipped, but next.md still listed them as
open work. Move both into the Landed paragraph, drop the delivered P0
account-creation on-ramp row (its "Tag @ViditBot" half stays covered by the
bot rows), and restate what's left: bot, backfill upgrades, value layer,
legal + release gates.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: vidit-admin <admin@vidit.app>
@vidit-admin vidit-admin added the no-docs-needed Exempts the docs-pairing CI check (PR with no docs/ + planning/ impact) label Jul 16, 2026
…nd proof counts)

Add an admin-only Detection quality panel surfacing the machine-extraction
pipeline's health. New read-only GET /admin/detection-stats returns:

- reject-rate: the share of machine detections (detected_from_url set) an
  owner closed straight out of `detected` (status='closed',
  before_closed_status='detected') without ever geolocating them, over all
  machine detections. A detection vouched into `geolocated`, or still
  awaiting review, is not a reject.
- pending quality counts over the live `detected` queue (deleted_at IS NULL):
  drafts missing a source media, a proof-role image, or a source_url, the
  pieces the geolocate floor will demand.

Business logic in services/admin.detection_quality_stats (two grouped
count(*) FILTER aggregates), a Pydantic AdminDetectionStatsRead, admin-only
auth like the other admin endpoints. Frontend adds a DetectionStatsPanel to
the admin page, composed from existing ui primitives with the exact
definition documented inline and in docs/api.md. Six backend tests pin the
definitions; api-types.ts regenerated.

Also moves two descoped v0.4 backfill rows (note-tweet.js long-form bodies,
archive authenticity sample-check) to Unscheduled candidates, and deletes
the now-shipped reject-rate tracker row.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: vidit-admin <admin@vidit.app>
@vidit-admin vidit-admin removed the no-docs-needed Exempts the docs-pairing CI check (PR with no docs/ + planning/ impact) label Jul 16, 2026
@vidit-admin vidit-admin changed the title docs(planning): sync v0.4 tracker with the shipped on-ramp and ingest-contract suite feat(admin): detection quality metrics (reject rate, missing source and proof counts) Jul 16, 2026
vidit-admin and others added 2 commits July 16, 2026 11:12
Comment-only: swap the clause-joining dash for a comma. No behaviour change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: vidit-admin <admin@vidit.app>
Triple-review follow-up on GET /admin/detection-stats.

- Reject-rate now counts a machine detection dismissed while still a draft
  whichever door it left through: owner close off `detected` OR soft-delete
  while still `detected`. A soft-deleted `geolocated` row stays a non-reject
  (vouched before removal). Mirrors services/detection._reimportable.
- Pending-queue aggregate now filters `detected_from_url IS NOT NULL` so the
  query matches its "machine detections" label.
- Both aggregates exclude demo rows (`is_demo`) so seeded fixtures don't
  pollute the metric.
- Proof-image predicate now requires `media_type = "image"`, so a video in the
  proof slot still counts as missing a proof image.
- Schema docstring, docs/api.md, and the panel description restate the new
  definitions and note the hard-delete / account-cascade counting edges.
- DetectionStatsPanel: "No machine detections yet" hint when machine_total is
  0; the admin-dialect comment points at docs/design.md, not planning/.
- frontend/src/lib/admin.ts: DetectionStats aliases the generated
  AdminDetectionStatsRead instead of a hand-written interface.
- vulture_whitelist.py: whitelist all 7 schema fields.
- Tests pin the new semantics (soft-deleted draft is a reject, soft-deleted
  geolocated is not, demo row moves nothing, proof video counts as missing an
  image); planning/next.md durable-worker cell de-dashed; api-types
  regenerated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: vidit-admin <admin@vidit.app>
@vidit-admin
vidit-admin merged commit ece9adf into main Jul 16, 2026
12 checks passed
@vidit-admin
vidit-admin deleted the claude/main-update-next-steps-af29ad branch July 16, 2026 10:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant