feat: status contract — Skip(reason, kind), warnings in JSON, enrichment counters, resource_logger (zeeker 0.9.0) - #10
Merged
Merged
Conversation
…chment counters, resource_logger
Makes build output unambiguous for AI monitoring agents:
- New `zeeker.Skip(reason, kind)` exception (kinds: up_to_date, blocked,
disabled) lets fetch_data distinguish "up to date" from "proxy down".
Handled on sync + async paths, cached by AsyncExecutor so the
single-fetch-per-build guarantee holds (schema-check sample fetch and
parallel pre-warm both re-observe the same Skip without re-running
fetch_data). Skip during the schema check is not a schema error.
fragments_on_skip still runs fragments with context [] on a raised Skip.
- Skip detection now flows through a typed ValidationResult.skipped flag
set by the processor — the "No data returned" string-matching in
builder._build_resource_outcome is gone. Returning [] still maps to a
skip with kind "up_to_date" and no reason.
- New `zeeker build --fail-on-blocked`: exit 1 when any resource skipped
with kind "blocked" (documented in the exit-codes section).
- ResourceOutcome.warnings + BuildReport.build_warnings now carry
ValidationResult warnings into --json / --progress-file payloads;
plain mode streams WARN[resource]/WARN[build] lines, rich mode shows a
warnings count in Notes and lists them when verbose.
- Optional module-level __zeeker_report__ dict ({"updated": 50, ...,
"notes": "..."}) read leniently after fetch_data and cleared per build;
surfaces as ResourceOutcome.extra_counts/notes, appended to success/skip
lines and aggregated into the SUMMARY footer — enrichment pipelines that
UPDATE rows are no longer invisible.
- zeeker-common: new zeeker_common.buildlog.resource_logger(name) —
prefix-consistent info/warn/error/done/aborted/skipped logging (pure
stdlib, stdout/stderr split) with best-effort JSONL sink via
ZEEKER_BUILDLOG_JSONL; lazily exported from the package root.
- Version bumps: zeeker 0.8.0 -> 0.9.0 (top-level __version__ synced),
zeeker-common 0.1.0 -> 0.2.0; uv.lock refreshed.
- Tests: 43 new (packages/zeeker/tests/test_status_contract.py,
packages/zeeker-common/tests/test_buildlog.py); full suite green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cuh5eJDfVowgpp22s7siE1
…er, count grammar
Reviewer findings addressed:
- Rich markup injection (helpers.py): skip reasons, error messages,
__zeeker_report__ count keys, warnings, fatal/FTS errors, post-hook
output, and tracebacks are now markup-escaped in TTY mode. Previously
Skip("proxy down [/socks5]") raised MarkupError inside the progress
callback (which runs inside build_database's try), converting a healthy
build into a fatal exit-2 error and aborting remaining resources;
bracketed tokens like [socks5h://...] were silently swallowed as style
tags. End-to-end regression test drives the TTY callback through a
real build.
- Blocked/disabled skips no longer advance _zeeker_updates.last_updated
(builder.py). Skip(kind="blocked") means "could not even check the
source"; bumping the freshness marker made time-based incremental
resources permanently miss anything published during an outage.
up_to_date skips (raised or returned-[]) still bump it, matching the
legacy returned-[] behavior.
- resource_logger count grammar (zeeker-common buildlog): done()/aborted()
now render noun-first ("done — 3 new, 2 skipped, 0 failed") to match
the grammar the data repos' Build Monitoring Guides already parse
("(\d+) new"), instead of "new=3" which would have silently broken
every monitoring regex on migration. Underscores render as spaces;
the JSONL sink keeps raw keys.
- sync_retry/async_retry no longer retry Skip (zeeker-common retry):
tenacity's default predicate retried a declared skip with 2-10s
backoff and resurfaced it as RetryError -> resource FAILED. Skip is
excluded by class name across the MRO (zeeker-common cannot import
zeeker), so shimmed Skip classes pass through too.
- Skip raised from fetch_fragments_data is now honored (processor.py):
it gracefully skips just the fragments phase instead of flipping the
resource to "failed" and the build to exit 1. Skip in transform_data
remains out of contract and is documented as such.
- __zeeker_report__ is consumed on every exit path (processor.py
try/finally): counters set before a fetch_data crash now surface on
the failed outcome and in --json, and counters set during the
fragments phase (fetch_fragments_data / transform_data) merge into
the outcome (summing on key collision) instead of being dropped.
- TTY --verbose no longer prints every warning twice (cli.py):
echo_warnings is skipped when _emit_rich's verbose loop lists the
same warnings under the summary table.
- Docs: CLAUDE.md + Skip docstring now state the minimum version
(zeeker>=0.9.0) for `from zeeker import Skip`, the try/except
ImportError shim pattern for repos pinned to older zeeker, the
retry-decorator interaction, where Skip is honored, and the
blocked/disabled freshness-marker semantics.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cuh5eJDfVowgpp22s7siE1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A logging audit across all five data projects found the root ambiguity that every repo's "Build Monitoring Guide" exists to work around: a resource can only raise (FAIL) or return a list (empty = SKIP, rendered "no data returned"), so "up to date" and "my proxy is down" are indistinguishable;
--jsonomits warnings entirely; and enrichment pipelines that update thousands of rows report 0 records. AI monitors were left inferring failure from run duration.The contract
1.
from zeeker import Skip—raise Skip("TAILSCALE_PROXY unset — proxy required", kind="blocked")(kinds:up_to_date,blocked,disabled). Caught on every fetch path (sync, async, schema-check sample,--parallelpre-warm — the Skip is cached sofetch_datastill runs exactly once). Renders as[SKIP] name <reason> (blocked) (0.1s), lands in--json/--progress-fileasskip_reason/skip_kind. Returning[]still works unchanged (up_to_date, "no data returned"). New--fail-on-blockedflag exits 1 on blocked skips (default off). The string-matched skip detection is gone — status is typed end-to-end.2. Warnings reach machine-readable output.
ResourceOutcome.warnings+BuildReport.build_warnings(S3-sync failures included) now appear in the JSON payload; plain mode streamsWARN[resource]:/WARN[build]:lines.3.
__zeeker_report__enrichment counters. A resource sets a module-level dict ({"updated": 50, "notes": "phase2 drained 25"}); the processor consumes it defensively on every outcome path (success/skip/crash, try/finally) intoextra_counts— so a "skipped" resource that did 50 rows of enrichment showsupdated=50on its line, in the SUMMARY footer, and in JSON.4.
zeeker_common.buildlog.resource_logger(name)— stdlib-only logger emitting the exact line grammar the data repos adopted this week ({name}: done — k=v,{name}: ABORTED (reason) — …to stderr,{name}: SKIPPED (reason)), with optional JSONL mirroring viaZEEKER_BUILDLOG_JSONL.Versions: zeeker 0.8.0 → 0.9.0 (also fixes
__init__.__version__which was desynced at 0.6.0), zeeker-common 0.1.0 → 0.2.0.Review process
Two adversarial reviewers (correctness; API-design/compat) produced 10 findings — all verified real, all fixed, notably:
[socks5h://…]raisedMarkupErrorinside the progress callback, turning a healthy build into a fatal exit-2. All user-controlled strings in TTY rendering are now escaped._zeeker_updates.last_updatedwas bumped during outages, silently losing data for time-based incremental resources. Blocked/disabled skips no longer touch it; up_to_date still does.zeeker_common.retrynow passes Skip through instead of burning backoff and resurfacing it asRetryError→ FAILED.zeeker>=0.9.0requirement and atry/except ImportErrorshim pattern for repos on older versions.Tests
uv run pytest: 447 passed, 5 skipped (pre-existing), coverage 78.29%. New suites cover Skip on all paths, blocked exit codes, JSON schema additions, counter consumption/reset/crash-survival, markup safety, and buildlog grammar.🤖 Generated with Claude Code
https://claude.ai/code/session_01Cuh5eJDfVowgpp22s7siE1
Generated by Claude Code