feat(admin): detection quality metrics (reject rate, missing source and proof counts) - #137
Merged
Merged
Conversation
…-contract suite The archive-upload on-ramp (POST /events/import-archive, in-browser strip, /submit bulk mode, confirm-registration landing) and the per-typology ingest-contract test suite both shipped, but next.md still listed them as open work. Move both into the Landed paragraph, drop the delivered P0 account-creation on-ramp row (its "Tag @ViditBot" half stays covered by the bot rows), and restate what's left: bot, backfill upgrades, value layer, legal + release gates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: vidit-admin <admin@vidit.app>
…nd proof counts) Add an admin-only Detection quality panel surfacing the machine-extraction pipeline's health. New read-only GET /admin/detection-stats returns: - reject-rate: the share of machine detections (detected_from_url set) an owner closed straight out of `detected` (status='closed', before_closed_status='detected') without ever geolocating them, over all machine detections. A detection vouched into `geolocated`, or still awaiting review, is not a reject. - pending quality counts over the live `detected` queue (deleted_at IS NULL): drafts missing a source media, a proof-role image, or a source_url, the pieces the geolocate floor will demand. Business logic in services/admin.detection_quality_stats (two grouped count(*) FILTER aggregates), a Pydantic AdminDetectionStatsRead, admin-only auth like the other admin endpoints. Frontend adds a DetectionStatsPanel to the admin page, composed from existing ui primitives with the exact definition documented inline and in docs/api.md. Six backend tests pin the definitions; api-types.ts regenerated. Also moves two descoped v0.4 backfill rows (note-tweet.js long-form bodies, archive authenticity sample-check) to Unscheduled candidates, and deletes the now-shipped reject-rate tracker row. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: vidit-admin <admin@vidit.app>
Comment-only: swap the clause-joining dash for a comma. No behaviour change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: vidit-admin <admin@vidit.app>
Triple-review follow-up on GET /admin/detection-stats. - Reject-rate now counts a machine detection dismissed while still a draft whichever door it left through: owner close off `detected` OR soft-delete while still `detected`. A soft-deleted `geolocated` row stays a non-reject (vouched before removal). Mirrors services/detection._reimportable. - Pending-queue aggregate now filters `detected_from_url IS NOT NULL` so the query matches its "machine detections" label. - Both aggregates exclude demo rows (`is_demo`) so seeded fixtures don't pollute the metric. - Proof-image predicate now requires `media_type = "image"`, so a video in the proof slot still counts as missing a proof image. - Schema docstring, docs/api.md, and the panel description restate the new definitions and note the hard-delete / account-cascade counting edges. - DetectionStatsPanel: "No machine detections yet" hint when machine_total is 0; the admin-dialect comment points at docs/design.md, not planning/. - frontend/src/lib/admin.ts: DetectionStats aliases the generated AdminDetectionStatsRead instead of a hand-written interface. - vulture_whitelist.py: whitelist all 7 schema fields. - Tests pin the new semantics (soft-deleted draft is a reject, soft-deleted geolocated is not, demo row moves nothing, proof video counts as missing an image); planning/next.md durable-worker cell de-dashed; api-types regenerated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: vidit-admin <admin@vidit.app>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two things land here: an admin-only detection-quality feature and the tracker upkeep around it.
Feature: admin detection-quality metrics
A new Detection quality panel in the admin page, admin-only, surfacing the health of the machine-extraction pipeline. Backed by a read-only
GET /admin/detection-stats(🛡️).Reject-rate. The share of machine detections dismissed while still a draft, whichever door they left through. A machine detection is a row imported from X (the archive backfill or the bot), discriminated by
detected_from_urlbeing set (a human submit carriesnull); demo rows (is_demo) are excluded. A detection counts as "rejected" if either an owner closed it straight out ofdetected(status = "closed"withbefore_closed_status = "detected") or an admin soft-deleted it while it was stilldetected(deleted_atset withstatus = "detected"). This matches theservices/detection._reimportabledismissal semantics, where soft-delete and owner close are the same judged-and-thrown-out shape. A detection vouched intogeolocated(even one later soft-deleted, it was vouched before removal), or one still awaiting review, is not a reject.reject_rate = machine_rejected / machine_totalas a 0..1 ratio (0 when there are no machine detections), counted over every non-demo machine row (soft-deleted or not, since the metric measures what the pipeline produced). Two accepted counting edges, both preferring over-counting dismissals: an owner hard-delete drops the row from both counts, and an account-departure cascade soft-delete counts that account's pending drafts as rejects.Pending quality counts. Over the live
detectedqueue (deleted_at IS NULL, machine rows only, demo excluded), three cheap signals count drafts missing a piece the geolocate floor will demand: no source media, no proof-role image (a video in the proof slot still lacks an image), nosource_url.Shape. Business logic in
services/admin.detection_quality_stats(two groupedcount(*) FILTER (WHERE …)aggregates), a PydanticAdminDetectionStatsRead, admin-only auth like the other admin endpoints. The frontend addsDetectionStatsPanelcomposed from existing ui primitives (admin surfaces inline their markup, no new shared primitive), with the exact definitions documented in the card and indocs/api.md. Backend tests pin the definitions (a closed-from-detected row is a reject, a soft-deleted draft is too, a geolocated one is not even once soft-deleted, a human submit moves neither total, a demo row moves nothing, and the pending counts flag each missing piece, a proof video counting as a missing image, while excluding soft-deleted rows).api-types.tsregenerated so the drift gate stays green.Tracker sync (already on the branch)
The earlier commit synced
planning/next.mdwith work shipped onmain: the ingest-contract test suite (#136) folded into the Landed paragraph, and the P0 archive on-ramp row deleted (its "Tag @ViditBot" half stays covered by the Bot rows).Tracker moves
note-tweet.jslong-form bodies and Archive authenticity sample-check. Not needed for v0.4; promoted back if a real archive needs them. The v0.4 "What's left" prose is restated to match.Gates
Backend (
backend/):ruff check .,ruff format --check .,mypy app(80),vulture,pytest(831 passed) — all green. Frontend (frontend/):tsc --noEmit,eslint .(0 errors),vitest(179),next build— all green. Repo rootmake hygiene(jscpd, knip, vulture, palette-coverage) — green.api-types.tsregenerated viamake gen-api-types.🤖 Generated with Claude Code