Skip to content

feat(dx): minimal install, reproducible CI, portable capture artifacts - #23

Merged
jeffbrennan merged 2 commits into
mainfrom
feat/developer-experience-brief-05
Sep 13, 2026
Merged

jeffbrennan merged 2 commits into
mainfrom
feat/developer-experience-brief-05

Conversation

@jeffbrennan

Copy link
Copy Markdown
Owner

Implements brief 05 (plans/improvements-2026-09/05-developer-experience.md) across all three increments.

Increment 1 — minimal install and CLI

  • Base install (polars, pydantic, typer) parses, analyzes, and runs sparkparse --help/--version/parse/analyze without PySpark or Dash. capture/connect load lazily via PEP 562; common.get_spark imports PySpark inside the function; app.viz imports the dashboard at the command boundary.
  • Version is derived from package metadata instead of a hardcoded string.
  • timeit writes to logging/stderr, so analyze stdout stays parseable JSON.
  • Log dir is a positional argument with a --log-dir alias; missing local paths raise a clear BadParameter.

Increment 2 — repeatable development

  • CI runs on synchronize with concurrency cancellation and UV_FROZEN=1.
  • New packaging job builds the wheel once and smoke-tests base / [viz] / [spark] installs in fresh environments (base asserts no PySpark/Dash import).
  • databricks.yml wheel path parameterized through a bundle variable; just build-wheel builds it.
  • README: extras, canonical CLI usage, dependency-update flow, and artifact docs; stale TODO claims corrected.

Increment 3 — portable capture artifact/report

  • New sparkparse/artifact.py: directory artifacts (manifest.json + Arrow dag/combined tables) plus single-file JSON, with round-trip including typed empty frames.
  • capture_context(..., artifact_path=...) persists the result (and viz report.html) on both success and failure; a write failure marks the capture partial and propagates under capture_errors="raise" without masking the workload exception.
  • New sparkparse/dataset.py: server-side cached access to a raw-log dir, an artifact, or an in-memory result.
  • init_dashboard(source) accepts any of those. Summary aggregates server-side (browser no longer receives the raw task table), the stage timeline is precomputed, tables paginate, and unavailable panels are listed with their reasons instead of rendering blank.

Validation

  • just ci (ruff check/format, pyrefly, pytest): 447 passed, 2 skipped.
  • Fresh wheel installs verified locally for base (no PySpark/Dash) and [viz]; CI covers [spark].
  • New tests: test_artifact.py, test_dataset.py, test_cli.py, test_home_page.py, test_dashboard_summary.py, plus failed-capture artifact coverage in test_capture_contract.py.

Not covered

The brief's manual Databricks serverless smoke procedure (Free Edition) was not run; that remains an outstanding live-validation item.

Implement brief 05 across packaging, developer experience, and reporting.

- base install (polars/pydantic/typer) parses, analyzes, and runs the CLI
  without PySpark or Dash; capture/connect import lazily, version comes
  from package metadata, and timing logs go to stderr
- accept --log-dir as an alias, validate missing paths, and add --artifact
- CI runs on synchronize with concurrency cancellation and frozen installs;
  a packaging job smoke-tests base/[viz]/[spark] wheel installs
- parameterize the Databricks bundle wheel path; update README usage
- add portable capture artifacts (manifest + Arrow tables) that the
  dashboard opens directly, with server-side frame caching and bounded,
  paginated payloads
Importing the capture submodule rebound the package attribute to the module,
shadowing the lazily-resolved decorator. Make capture.py free of a top-level
PySpark import (import it inside the session-building functions) and bind
capture/connect eagerly at package import. Base install still never imports
PySpark or Dash.
@jeffbrennan
jeffbrennan merged commit 0e0a079 into main Sep 13, 2026
4 checks passed
@jeffbrennan
jeffbrennan deleted the feat/developer-experience-brief-05 branch September 13, 2026 01:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant