Skip to content

Safe capture lifecycle and capability contract - #19

Merged
jeffbrennan merged 4 commits into
mainfrom
feat/brief-01-capture-capabilities
Sep 12, 2026
Merged

jeffbrennan merged 4 commits into
mainfrom
feat/brief-01-capture-capabilities

Conversation

@jeffbrennan

Copy link
Copy Markdown
Owner

Implements plans/improvements-2026-09/01-capture-and-capabilities.md.

  • capture no longer stops or recreates a supplied session — borrowed classic sessions must have event logging enabled before the workload starts; own_session=True opts into a session sparkparse owns and shuts down before parsing
  • explicit capability contract — CaptureResult carries dag, combined, metadata, capabilities, and diagnostics; unavailable Connect task/stage telemetry is reported as unavailable rather than as zeroes, and analysis/history keep those totals null
  • public result surface — cap.result, cap.dfs, cap.analysis, cap.last_record, cap.triggered_alerts; _parsed_logs is gone with no legacy alias
  • backend selection — backend="classic" | "connect" | "event_log", the last for post-run ingestion with no session; borrowed captures select the log by application ID
  • capture error policy — failures raise by default, capture_errors="record" keeps diagnostics and a partial result; user workload exceptions always take precedence, and failed captures retain their temp logs
  • action="viz" returns an in-memory HTML report in cap.report on either backend instead of launching a server; capture-result JSON uses Arrow IPC so schemas, empty tables, and nested metrics survive a round trip
  • tests: tests/test_capture_contract.py (new, 290 lines), plus history and capture additions

Smoke-tested on Databricks Free serverless (run 566565689796128, 2026-09-12) alongside the local suite. That run was Connect-only — classic shutdown and log-flushing behavior is covered locally. A timezone inference failure in history reads under newer Polars surfaced there and is fixed with explicit UTC parsing and an offset-normalization regression test.

@jeffbrennan
jeffbrennan merged commit 4998d10 into main Sep 12, 2026
2 checks passed
@jeffbrennan
jeffbrennan deleted the feat/brief-01-capture-capabilities branch September 12, 2026 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant