Skip to content

Compile Python scripts and notebooks into Texera workflows #8319

Description

@carloea2

Feature Summary

Texera users currently have to rewrite Python programs manually to execute them as workflows. Add a compiler path that accepts a .py file or executable Jupyter notebook code cells and produces an ordinary Texera workflow of Python UDF operators.

The initial scope groups complete top-level statements. Supported compound statements such as if, for, while, try, and with remain single placement units; their bodies are not distributed. Unsupported Python semantics return source-located diagnostics. Compilation performs static analysis and does not execute user code.

Proposed Solution or Design

.py or .ipynb code cells
          |
          v
ANALYZE -> GROUP -> VERIFY -> BUILD
          |
          v
ordinary Texera workflow -> standard Amber execution
Layer Responsibility
Compiler Parse source, derive statement dependencies, choose legal groups, verify boundaries, and emit workflow JSON
PyTexera runtime Import and export selected Python values through a Cloudpickle envelope
Texera integration Invoke the compiler, report safe diagnostics, create the workflow, and execute standard Python UDF operators

The default grouping strategy targets min(statement_count, ceil(sqrt(physical LOC))) contiguous groups. Every proposal must remain acyclic and satisfy dependency, local reconstruction, and boundary transport checks. A one-statement-per-operator strategy remains available as a diagnostic baseline.

Initial scope:

Supported Deferred with a source-located diagnostic
Imports, assignments, expressions, and ordinary calls Function and class definitions
Complete if, for, while, try, and with statements Lambdas, async constructs, global, and nonlocal
Python files and executable notebook code cells Wildcard and relative imports
Transport of downstream-required values Intrastatement and control-body decomposition

Acceptance criteria:

  • .py and .ipynb inputs compile through the same path.
  • Generated JSON imports as a normal Texera workflow.
  • Generated UDFs execute in an Amber integration test.
  • Boundary transport preserves required values, aliases, cycles, and field presence.
  • Known unsupported source returns an HTTP 422 response with a source location.
  • Unexpected failures return a generic HTTP 500 response while details remain server-side.
  • The frontend displays compiler diagnostics and creates a workflow on success.
  • The deployed service contains the compiler runtime and does not depend on a repository checkout.

Design discussion: #8160

Pull request plan

Each row is independently reviewable. Compiler rows are stacked only where the later layer imports the earlier one.

# Scope Approx. LOC Depends on Status
1 PyAmber channel-marker coordination 200 Withdrawn
2 PyTexera Cloudpickle transport runtime 1,050 Withdrawn
3 Source-exact Action forest and inventory 3,300 Withdrawn
4 Carrier model and dependency graph 700 #3 Withdrawn
5 Lexical scopes and control traversal 2,800 #4 Withdrawn
6 Binding flow and statement state 900 #5 Planned
7 Points-to, alias, import, and transfer analysis 2,800 #6 Planned
8 Effects, liveness, and linearization 2,500 #7 Planned
9 Statement atoms, admissions, coloring, and placement views 2,800 #8 Planned
10 Realization batching, profiles, and selection 2,200 #9 Planned
11 Value, import, state, and world-order realization providers 2,500 #10, #2 Planned
12 Compiler session, configuration, checker, and physical plan 2,500 #11 Planned
13 UDF composition, rendering, and local executor 2,500 #12 Planned
14 Public API, CLI, workflow export, and end-to-end compiler tests 2,000 #13 Planned
15 Migration-service packaging and sanitized HTTP diagnostics 1,200 #14 Planned
16 Compiler tab, file input, workflow creation, and UI diagnostics 1,000 #15 Planned

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions