Repository navigation
feat(dx): minimal install, reproducible CI, portable capture artifacts - #23
Merged
Merged
Conversation
Implement brief 05 across packaging, developer experience, and reporting. - base install (polars/pydantic/typer) parses, analyzes, and runs the CLI without PySpark or Dash; capture/connect import lazily, version comes from package metadata, and timing logs go to stderr - accept --log-dir as an alias, validate missing paths, and add --artifact - CI runs on synchronize with concurrency cancellation and frozen installs; a packaging job smoke-tests base/[viz]/[spark] wheel installs - parameterize the Databricks bundle wheel path; update README usage - add portable capture artifacts (manifest + Arrow tables) that the dashboard opens directly, with server-side frame caching and bounded, paginated payloads
Importing the capture submodule rebound the package attribute to the module, shadowing the lazily-resolved decorator. Make capture.py free of a top-level PySpark import (import it inside the session-building functions) and bind capture/connect eagerly at package import. Base install still never imports PySpark or Dash.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements brief 05 (
plans/improvements-2026-09/05-developer-experience.md) across all three increments.Increment 1 — minimal install and CLI
polars,pydantic,typer) parses, analyzes, and runssparkparse --help/--version/parse/analyzewithout PySpark or Dash.capture/connectload lazily via PEP 562;common.get_sparkimports PySpark inside the function;app.vizimports the dashboard at the command boundary.timeitwrites to logging/stderr, soanalyzestdout stays parseable JSON.--log-diralias; missing local paths raise a clearBadParameter.Increment 2 — repeatable development
synchronizewith concurrency cancellation andUV_FROZEN=1.[viz]/[spark]installs in fresh environments (base asserts no PySpark/Dash import).databricks.ymlwheel path parameterized through a bundle variable;just build-wheelbuilds it.Increment 3 — portable capture artifact/report
sparkparse/artifact.py: directory artifacts (manifest.json+ Arrowdag/combinedtables) plus single-file JSON, with round-trip including typed empty frames.capture_context(..., artifact_path=...)persists the result (and vizreport.html) on both success and failure; a write failure marks the capture partial and propagates undercapture_errors="raise"without masking the workload exception.sparkparse/dataset.py: server-side cached access to a raw-log dir, an artifact, or an in-memory result.init_dashboard(source)accepts any of those. Summary aggregates server-side (browser no longer receives the raw task table), the stage timeline is precomputed, tables paginate, and unavailable panels are listed with their reasons instead of rendering blank.Validation
just ci(ruff check/format, pyrefly, pytest): 447 passed, 2 skipped.[viz]; CI covers[spark].test_artifact.py,test_dataset.py,test_cli.py,test_home_page.py,test_dashboard_summary.py, plus failed-capture artifact coverage intest_capture_contract.py.Not covered
The brief's manual Databricks serverless smoke procedure (Free Edition) was not run; that remains an outstanding live-validation item.