perf(gfql/cypher): single-alias predicate pushdown in the row pipeline (L4)#1702
Open
lmeyerov wants to merge 1 commit into
Open
perf(gfql/cypher): single-alias predicate pushdown in the row pipeline (L4)#1702lmeyerov wants to merge 1 commit into
lmeyerov wants to merge 1 commit into
Conversation
…e (L4)
A disjunctive / searchAny WHERE collapses the whole MATCH onto the row pipeline,
which materializes the full (a)-[e]->(b) binding table (O(E) joins) BEFORE
filtering. Add a lowering pass that attaches conjuncts referencing exactly ONE
alias (and standalone searchAny ops) to rows(alias_prefilters={alias:[spec]}) as
an ADVISORY HINT; the binding builder pre-filters each alias FRAME before the join.
Redundant-filter safety: the hint is additive — the post-join where_rows /
search_any ops are LEFT IN PLACE. Engines whose row builder honors the hint
(pandas / cuDF via RowPipelineMixin) pre-filter and win; engines that ignore it
(polars' native rows_binding_ops_polars, the shortestPath scalar builder) run the
unchanged post-filter and stay correct. A pushed conjunct is a literal
sub-conjunct of the retained AND, so the pre-filter can only remove rows the
post-filter also removes — never a result change, on any engine. Application
reuses the SAME evaluator (_gfql_eval_string_expr + _gfql_bool_mask) and the SAME
search_any_mask kernel.
Guards (skip pushing a conjunct/op): OPTIONAL / reentry / row-guard / procedure /
seed-row / multi-graph programs; shortestPath scalar bindings; multihop edge
aliases; and any conjunct naming an internal __gfql_/__cypher_ column (EXISTS /
searchAny markers + reentry carries read falsely single-alias).
Results (pandas, seed=7, base = pass-disabled):
search_edges @100k 2.9x @1m 4.62x (7.1s->1.5s)
panel_filters @100k 1.32x @1m 1.47x (24.9s->17.0s)
combined 1.97x ; search_nodes 1.1x ; exists_prune no-op
Byte-identical result multisets. cuDF parity validated (dgx RAPIDS 26.02, 17/17).
Full cypher+row+conformance suite green (1942 passed; the 4 conformance reds are
pre-existing on the clean base, verified).
Scope: polars/polars-gpu already NIE the multi-node `rows` op for these
disjunctive-WHERE row queries (pre-existing), so this accelerates the pandas +
cuDF row-pipeline lanes; polars correctness is preserved via the retained
post-filter.
New: graphistry/compute/gfql/cypher/row_pushdown.py + test_row_pushdown.py (17
tests: parity + guards + pass internals; cuDF lane via TEST_CUDF).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
cd6f8ae to
3b6b855
Compare
f670c9e to
a8ab8f2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A disjunctive /
searchAnyWHERE collapses the whole MATCH onto the row pipeline, which materializes the full(a)-[e]->(b)binding table (O(E) joins) before filtering. This adds a lowering pass that attaches conjuncts referencing exactly one alias (and standalonesearchAnyops) torows(alias_prefilters={alias:[spec]})as an advisory hint; the binding builder pre-filters each alias frame before the join.Redundant-filter safety (why it's correct on every engine)
The hint is additive — the post-join
where_rows/search_anyops are left in place:RowPipelineMixin) pre-filter each alias frame before the binding join and win.rows_binding_ops_polars, the shortestPath scalar builder) run the unchanged post-filter and stay correct.A pushed conjunct is a literal sub-conjunct of the retained AND, so the pre-filter can only remove rows the post-filter also removes — never a result change, on any engine (present or future). Application reuses the SAME evaluator (
_gfql_eval_string_expr+_gfql_bool_mask) and the SAMEsearch_any_maskkernel, so semantics (incl. 3-valued NULL) are byte-identical.Guards (skip pushing a conjunct/op)
OPTIONAL / reentry / row-guard / procedure / seed-row / multi-graph programs; shortestPath scalar bindings; multihop edge aliases; and any conjunct naming an internal
__gfql_/__cypher_column (EXISTS / searchAny markers + reentry carries read falsely single-alias).Results (pandas, seed=7; base = pass-disabled)
Byte-identical result multisets.
search_edgeskeeps its full win (post-filter search on the small post-join table is cheap);panel_filtersgives back part of the win because the retainedwhere_rowsre-evaluates string predicates on the post-join table (a documented follow-on reclaims it via an executor handshake).Scope
polars/polars-gpu already NIE the multi-node
rowsop for these disjunctive-WHERE row queries (pre-existing), so this accelerates the pandas + cuDF row-pipeline lanes; polars correctness is preserved by the retained post-filter.Tests
test_row_pushdown.py: 17 tests — parity pre/post pass (pandas + cuDF viaTEST_CUDF) on edge/node search, panel, undirected, combined; guard no-push on EXISTS-marker / multi-alias / shortestPath; pass internals.regex_unicode_trick[ceil/floor/round-filter],categorical_searchany) are pre-existing — verified failing on the clean base commit and with the pass disabled.🤖 Generated with Claude Code