Feature Summary
Texera users currently have to rewrite Python programs manually to execute them as workflows. Add a compiler path that accepts a .py file or executable Jupyter notebook code cells and produces an ordinary Texera workflow of Python UDF operators.
The initial scope groups complete top-level statements. Supported compound statements such as if, for, while, try, and with remain single placement units; their bodies are not distributed. Unsupported Python semantics return source-located diagnostics. Compilation performs static analysis and does not execute user code.
Proposed Solution or Design
.py or .ipynb code cells
|
v
ANALYZE -> GROUP -> VERIFY -> BUILD
|
v
ordinary Texera workflow -> standard Amber execution
| Layer |
Responsibility |
| Compiler |
Parse source, derive statement dependencies, choose legal groups, verify boundaries, and emit workflow JSON |
| PyTexera runtime |
Import and export selected Python values through a Cloudpickle envelope |
| Texera integration |
Invoke the compiler, report safe diagnostics, create the workflow, and execute standard Python UDF operators |
The default grouping strategy targets min(statement_count, ceil(sqrt(physical LOC))) contiguous groups. Every proposal must remain acyclic and satisfy dependency, local reconstruction, and boundary transport checks. A one-statement-per-operator strategy remains available as a diagnostic baseline.
Initial scope:
| Supported |
Deferred with a source-located diagnostic |
| Imports, assignments, expressions, and ordinary calls |
Function and class definitions |
Complete if, for, while, try, and with statements |
Lambdas, async constructs, global, and nonlocal |
| Python files and executable notebook code cells |
Wildcard and relative imports |
| Transport of downstream-required values |
Intrastatement and control-body decomposition |
Acceptance criteria:
.py and .ipynb inputs compile through the same path.
- Generated JSON imports as a normal Texera workflow.
- Generated UDFs execute in an Amber integration test.
- Boundary transport preserves required values, aliases, cycles, and field presence.
- Known unsupported source returns an HTTP 422 response with a source location.
- Unexpected failures return a generic HTTP 500 response while details remain server-side.
- The frontend displays compiler diagnostics and creates a workflow on success.
- The deployed service contains the compiler runtime and does not depend on a repository checkout.
Design discussion: #8160
Pull request plan
Each row is independently reviewable. Compiler rows are stacked only where the later layer imports the earlier one.
| # |
Scope |
Approx. LOC |
Depends on |
Status |
| 1 |
PyAmber channel-marker coordination |
200 |
— |
Withdrawn |
| 2 |
PyTexera Cloudpickle transport runtime |
1,050 |
— |
Withdrawn |
| 3 |
Source-exact Action forest and inventory |
3,300 |
— |
Withdrawn |
| 4 |
Carrier model and dependency graph |
700 |
#3 |
Withdrawn |
| 5 |
Lexical scopes and control traversal |
2,800 |
#4 |
Withdrawn |
| 6 |
Binding flow and statement state |
900 |
#5 |
Planned |
| 7 |
Points-to, alias, import, and transfer analysis |
2,800 |
#6 |
Planned |
| 8 |
Effects, liveness, and linearization |
2,500 |
#7 |
Planned |
| 9 |
Statement atoms, admissions, coloring, and placement views |
2,800 |
#8 |
Planned |
| 10 |
Realization batching, profiles, and selection |
2,200 |
#9 |
Planned |
| 11 |
Value, import, state, and world-order realization providers |
2,500 |
#10, #2 |
Planned |
| 12 |
Compiler session, configuration, checker, and physical plan |
2,500 |
#11 |
Planned |
| 13 |
UDF composition, rendering, and local executor |
2,500 |
#12 |
Planned |
| 14 |
Public API, CLI, workflow export, and end-to-end compiler tests |
2,000 |
#13 |
Planned |
| 15 |
Migration-service packaging and sanitized HTTP diagnostics |
1,200 |
#14 |
Planned |
| 16 |
Compiler tab, file input, workflow creation, and UI diagnostics |
1,000 |
#15 |
Planned |
Feature Summary
Texera users currently have to rewrite Python programs manually to execute them as workflows. Add a compiler path that accepts a
.pyfile or executable Jupyter notebook code cells and produces an ordinary Texera workflow of Python UDF operators.The initial scope groups complete top-level statements. Supported compound statements such as
if,for,while,try, andwithremain single placement units; their bodies are not distributed. Unsupported Python semantics return source-located diagnostics. Compilation performs static analysis and does not execute user code.Proposed Solution or Design
The default grouping strategy targets
min(statement_count, ceil(sqrt(physical LOC)))contiguous groups. Every proposal must remain acyclic and satisfy dependency, local reconstruction, and boundary transport checks. A one-statement-per-operator strategy remains available as a diagnostic baseline.Initial scope:
if,for,while,try, andwithstatementsglobal, andnonlocalAcceptance criteria:
.pyand.ipynbinputs compile through the same path.Design discussion: #8160
Pull request plan
Each row is independently reviewable. Compiler rows are stacked only where the later layer imports the earlier one.