Skip to content

perf(gfql/cypher): single-alias predicate pushdown in the row pipeline (L4)#1702

Open
lmeyerov wants to merge 1 commit into
feat/gfql-index-degreesfrom
perf/gfql-row-pushdown
Open

perf(gfql/cypher): single-alias predicate pushdown in the row pipeline (L4)#1702
lmeyerov wants to merge 1 commit into
feat/gfql-index-degreesfrom
perf/gfql-row-pushdown

Conversation

@lmeyerov

@lmeyerov lmeyerov commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

What

A disjunctive / searchAny WHERE collapses the whole MATCH onto the row pipeline, which materializes the full (a)-[e]->(b) binding table (O(E) joins) before filtering. This adds a lowering pass that attaches conjuncts referencing exactly one alias (and standalone searchAny ops) to rows(alias_prefilters={alias:[spec]}) as an advisory hint; the binding builder pre-filters each alias frame before the join.

Redundant-filter safety (why it's correct on every engine)

The hint is additive — the post-join where_rows / search_any ops are left in place:

  • Engines whose row builder honors the hint (pandas / cuDF via RowPipelineMixin) pre-filter each alias frame before the binding join and win.
  • Engines that ignore it (polars' native rows_binding_ops_polars, the shortestPath scalar builder) run the unchanged post-filter and stay correct.

A pushed conjunct is a literal sub-conjunct of the retained AND, so the pre-filter can only remove rows the post-filter also removes — never a result change, on any engine (present or future). Application reuses the SAME evaluator (_gfql_eval_string_expr + _gfql_bool_mask) and the SAME search_any_mask kernel, so semantics (incl. 3-valued NULL) are byte-identical.

Guards (skip pushing a conjunct/op)

OPTIONAL / reentry / row-guard / procedure / seed-row / multi-graph programs; shortestPath scalar bindings; multihop edge aliases; and any conjunct naming an internal __gfql_/__cypher_ column (EXISTS / searchAny markers + reentry carries read falsely single-alias).

Results (pandas, seed=7; base = pass-disabled)

scenario @100k @1m
search_edges 2.9× 4.62× (7.1s→1.5s)
panel_filters 1.32× 1.47× (24.9s→17.0s)
combined 1.97×
search_nodes 1.1×
exists_prune no-op (EXISTS not pushable)

Byte-identical result multisets. search_edges keeps its full win (post-filter search on the small post-join table is cheap); panel_filters gives back part of the win because the retained where_rows re-evaluates string predicates on the post-join table (a documented follow-on reclaims it via an executor handshake).

Scope

polars/polars-gpu already NIE the multi-node rows op for these disjunctive-WHERE row queries (pre-existing), so this accelerates the pandas + cuDF row-pipeline lanes; polars correctness is preserved by the retained post-filter.

Tests

  • New test_row_pushdown.py: 17 tests — parity pre/post pass (pandas + cuDF via TEST_CUDF) on edge/node search, panel, undirected, combined; guard no-push on EXISTS-marker / multi-alias / shortestPath; pass internals.
  • cuDF lane validated on dgx (RAPIDS 26.02): 17/17 passed.
  • Full cypher+row+conformance suite: 1942 passed; the 4 conformance reds (regex_unicode_trick[ceil/floor/round-filter], categorical_searchany) are pre-existing — verified failing on the clean base commit and with the pass disabled.

🤖 Generated with Claude Code

…e (L4)

A disjunctive / searchAny WHERE collapses the whole MATCH onto the row pipeline,
which materializes the full (a)-[e]->(b) binding table (O(E) joins) BEFORE
filtering. Add a lowering pass that attaches conjuncts referencing exactly ONE
alias (and standalone searchAny ops) to rows(alias_prefilters={alias:[spec]}) as
an ADVISORY HINT; the binding builder pre-filters each alias FRAME before the join.

Redundant-filter safety: the hint is additive — the post-join where_rows /
search_any ops are LEFT IN PLACE. Engines whose row builder honors the hint
(pandas / cuDF via RowPipelineMixin) pre-filter and win; engines that ignore it
(polars' native rows_binding_ops_polars, the shortestPath scalar builder) run the
unchanged post-filter and stay correct. A pushed conjunct is a literal
sub-conjunct of the retained AND, so the pre-filter can only remove rows the
post-filter also removes — never a result change, on any engine. Application
reuses the SAME evaluator (_gfql_eval_string_expr + _gfql_bool_mask) and the SAME
search_any_mask kernel.

Guards (skip pushing a conjunct/op): OPTIONAL / reentry / row-guard / procedure /
seed-row / multi-graph programs; shortestPath scalar bindings; multihop edge
aliases; and any conjunct naming an internal __gfql_/__cypher_ column (EXISTS /
searchAny markers + reentry carries read falsely single-alias).

Results (pandas, seed=7, base = pass-disabled):
  search_edges   @100k 2.9x   @1m 4.62x (7.1s->1.5s)
  panel_filters  @100k 1.32x  @1m 1.47x (24.9s->17.0s)
  combined 1.97x ; search_nodes 1.1x ; exists_prune no-op
Byte-identical result multisets. cuDF parity validated (dgx RAPIDS 26.02, 17/17).
Full cypher+row+conformance suite green (1942 passed; the 4 conformance reds are
pre-existing on the clean base, verified).

Scope: polars/polars-gpu already NIE the multi-node `rows` op for these
disjunctive-WHERE row queries (pre-existing), so this accelerates the pandas +
cuDF row-pipeline lanes; polars correctness is preserved via the retained
post-filter.

New: graphistry/compute/gfql/cypher/row_pushdown.py + test_row_pushdown.py (17
tests: parity + guards + pass internals; cuDF lane via TEST_CUDF).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant