Skip to content

refactor(query): decouple logical optimization from physical planning - #20604

Open
forsaken628 wants to merge 4 commits into
databendlabs:mainfrom
forsaken628:refactor-mut
Open

forsaken628 wants to merge 4 commits into
databendlabs:mainfrom
forsaken628:refactor-mut

Conversation

@forsaken628

@forsaken628 forsaken628 commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Previously, a single SExpr representation and SExpr → SExpr optimizer interface tied frontend logical rewrites to CBO and physical-plan finalization. Logical semantics and execution-stage concerns shared the same
contract, making maintenance difficult and constraining how CBO could evolve to model physical implementations.

This PR separates those stages without changing the existing search and costing algorithms:

  • Keep the frontend optimizer pipeline on logical SExpr, and introduce a separate physical-planning entry that consumes SExpr and produces PExpr.
  • Share expression structure, traversal, and validation through a generic skeleton, while keeping logical and physical state and invalidation policies distinct.
  • Migrate physical rewrites and execution-plan builders to PExpr, with no reverse conversion to SExpr.
  • Prepare Mutation inputs and decorrelate INSERT WHEN subqueries before physical planning, rather than modifying logical structure after selection.
  • Handle physical EvalScalar cleanup independently and remove unused Memo collection from the logical pipeline.
  • Read lineage for planned statements from binding-time snapshots.

This establishes an independently evolvable physical representation. It does not yet introduce new physical implementation alternatives or change the CBO cost model.

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

AI assistance

  • AI usage: AI assistance was used for implementation, refactoring, review, and validation.
  • Responsible human: @forsaken628
  • The responsible human has read every line of this diff and can explain each change

This change is Reviewable

@github-actions github-actions Bot added the pr-refactor this PR changes the code base without new features or bugfix label Oct 1, 2026
Keep logical preparation on SExpr and extract selected implementations as an
independent PExpr tree. Migrate physical passes and recursive builders in place
without changing their algorithms or execution order.

Separate bound and planned query/mutation lifecycles, apply physical EvalScalar
cleanup directly, and retain mutation distribution policy on the physical tree.
Remove reverse expression conversion and obsolete logical-pipeline Memo capture.

Include lifecycle, expression-fork, cleanup, mutation and plan-golden regression
coverage with the final interfaces, without intermediate wrapper/pass designs.
Read lineage for planned queries and mutations only from binding-time snapshots.
Return no lineage when a planned statement has no snapshot, rather than
attempting to reconstruct SQL dependencies from an execution tree.

Keep direct extraction for bound statements and cover snapshot/no-snapshot
behavior for INSERT, CTAS, multi-table INSERT, UPDATE and DELETE.
@forsaken628
forsaken628 marked this pull request as ready for review October 1, 2026 15:32
@forsaken628
forsaken628 requested a review from sundy-li October 1, 2026 15:33

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-refactor this PR changes the code base without new features or bugfix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant