Skip to content

perf(gfql): real q1-q9 OLAP lowering on Polars#1714

Open
lmeyerov wants to merge 23 commits into
perf/gfql-row-pushdownfrom
perf/gfql-olap-real-q1q9
Open

perf(gfql): real q1-q9 OLAP lowering on Polars#1714
lmeyerov wants to merge 23 commits into
perf/gfql-row-pushdownfrom
perf/gfql-olap-real-q1q9

Conversation

@lmeyerov

@lmeyerov lmeyerov commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the real-GFQL graph-benchmark q1-q9 Polars path and the supporting OLAP lowering/row-pipeline optimizations needed to run it without dataframe shortcuts or untimed precompute.

Stack position: above #1702 (perf/gfql-row-pushdown) and below #1704 (bench/gfql-fair-matrix).

Validation

  • git diff --check origin/perf/gfql-row-pushdown...perf/gfql-olap-real-q1q9
  • python -m py_compile benchmarks/gfql/graph_benchmark_q1_q9_real.py graphistry/compute/gfql_unified.py graphistry/compute/gfql/cypher/lowering.py graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py
  • python -m pytest -q graphistry/tests/compute/gfql/test_engine_polars_binding_rows.py graphistry/tests/compute/gfql/test_engine_polars_row_pipeline.py graphistry/tests/compute/gfql/cypher/test_row_pushdown.py
  • python -m pytest -q graphistry/tests/compute/gfql/cypher/test_lowering.py -k '1712 or 1273 or t1 or t3 or t4 or graph_benchmark'
  • ./bin/typecheck.sh

DGX pressure evidence

Artifacts under plans/gfql-olap-optimization/:

  • dgx_t118_combined_polars.json
  • dgx_t119_combined_polars_confirm.json
  • dgx_t121_pokec_medium_combined.json

Graph-benchmark q1-q9 real GFQL Polars: parity OK across q1-q9. Performance is broadly stable vs T28; q5 remains a micro-query outlier to explain/optimize before final report claims.

Pokec medium native indexed traversal: row counts match prior #1658 artifact and GFQL pandas remains 8/8 faster than prior Memgraph medians.

Notes

Draft until CI and review-skill artifacts are refreshed for this exact PR scope.

@lmeyerov lmeyerov force-pushed the perf/gfql-olap-real-q1q9 branch 3 times, most recently from dc01f38 to c7b5c71 Compare July 8, 2026 08:55
@lmeyerov lmeyerov force-pushed the perf/gfql-row-pushdown branch from ed6e12c to 0a2e00a Compare July 8, 2026 08:55
@lmeyerov lmeyerov force-pushed the perf/gfql-olap-real-q1q9 branch from c7b5c71 to ea3fcd9 Compare July 8, 2026 09:01
@lmeyerov lmeyerov marked this pull request as ready for review July 8, 2026 09:58
@lmeyerov lmeyerov force-pushed the perf/gfql-row-pushdown branch from e2d693e to d23a5f4 Compare July 9, 2026 01:03
@lmeyerov lmeyerov force-pushed the perf/gfql-olap-real-q1q9 branch from ea3fcd9 to 36cf2ff Compare July 9, 2026 01:05
@lmeyerov lmeyerov force-pushed the perf/gfql-row-pushdown branch from d23a5f4 to 75193ba Compare July 9, 2026 01:47
@lmeyerov lmeyerov force-pushed the perf/gfql-olap-real-q1q9 branch from 36cf2ff to cc531c4 Compare July 9, 2026 01:47
@lmeyerov lmeyerov force-pushed the perf/gfql-row-pushdown branch from 75193ba to c91b86d Compare July 9, 2026 02:08
@lmeyerov lmeyerov force-pushed the perf/gfql-olap-real-q1q9 branch from cc531c4 to 180ad49 Compare July 9, 2026 02:09
@lmeyerov lmeyerov force-pushed the perf/gfql-row-pushdown branch from c91b86d to 5c2afb9 Compare July 9, 2026 02:41
@lmeyerov lmeyerov force-pushed the perf/gfql-olap-real-q1q9 branch from 180ad49 to f7cc095 Compare July 9, 2026 02:42
@lmeyerov lmeyerov force-pushed the perf/gfql-row-pushdown branch from 5c2afb9 to a40b01c Compare July 9, 2026 02:50
@lmeyerov lmeyerov force-pushed the perf/gfql-olap-real-q1q9 branch from f7cc095 to b09397f Compare July 9, 2026 02:50
@lmeyerov lmeyerov force-pushed the perf/gfql-row-pushdown branch from a40b01c to 1657638 Compare July 9, 2026 04:03
lmeyerov and others added 13 commits July 8, 2026 21:04
@lmeyerov @claude
fix(gfql): real-GFQL graph-bench count queries — polars count(*) broa…
8e3f0e7 
…dcast (#1707) + count(non-active node alias) routing (#1708)

Two correctness fixes that let graph-benchmark q1-q9 run as real GFQL Cypher
(no dataframe shortcuts) on pandas/cuDF, and remove a polars silent-wrong answer.

  `RETURN count(*)` (and any all-scalar-literal projection) returned constant 1
  instead of the true count. Cypher WITH/RETURN preserves row cardinality, but
  polars DataFrame.select of an all-scalar projection collapses to 1 row, so the
  synthetic __cypher_group__=1 for keyless count(*) made the downstream group_by
  count see 1 row. select_polars now broadcasts an all-scalar projection to the
  frame height via with_columns(...).select(names). +5 parity + 2 value-regression tests.

  `MATCH (a)-[e]->(b) RETURN b.id, count(a)` (graph-bench q1 in-degree) failed with
  "one MATCH source alias at a time" — the aggregate referenced the other endpoint
  alias, routing to the single-alias table instead of the bindings-row table.
  _lower_general_row_projection now forces the bindings source for count(<bare node
  alias>) (non-distinct) over a non-active pattern node alias — sound on the bindings
  table (count of matched paths per group = in-degree). Narrow: property/compound
  cross-source aggregates keep the conservative fail-fast. +3 tests (incl. cuDF).

Regression sweep (cypher/ + polars engine + row-pipeline): 2177 passed, 11 skipped, 15 xfailed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
@lmeyerov @claude
perf(gfql): chain edge-index attach without O(E) deep copy (#1670, #887)
58408d8 
Every chain()/Cypher call on a graph without an edge-id binding attached a
synthetic per-edge index via reset_index + rename, which deep-copied AND
block-consolidated the entire edge frame (~70ms @2m edges) — even for
node-only queries, and repeatedly per boundary-call sub-chain.

Now: shallow copy + assign the frame index as the id column — identical id
values (any index type), no O(E) data copy. The column is internal-only and
dropped on every exit path, so only uniqueness semantics matter (unchanged).

Measured @500k nodes / 2M edges (pandas):
  1-row point RETURN   95.7 -> 13.5 ms (7x)
  full-scan RETURN a   134  -> 55 ms  (2.4x)
  seeded 1-hop         192  -> ~170 ms
  polars               unchanged (own chain path)
Scaling (1-row RETURN): 15.6@50k/100@500k -> 6.8@50k/13.5@500k/42@1M.

Residual O(N/E) tail (backward-pass hydration merges, endpoints
reconciliation) remains tracked in #1670/#887; a hydration-skip variant was
evaluated and rejected (row-order semantics risk for ~1.5ms).

Suites: compute 4301 passed + gfql 3014 passed (9 pre-existing umap/dask
env failures, verified identical pre-fix via stash).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
@lmeyerov @claude
perf(gfql): hop edge-index attach without O(E) deep copy (#1670, #887)
b423d81 
Same fix as the chain.py edge-index attach (8765c0bd), applied to hop.py —
the traversal hot path. reset_index(drop=False) + rename deep-copied +
block-consolidated the whole post-filter edge frame (~80ms @2m edges) on
every hop, even seeded ones. Now a shallow copy + assigning the frame index
as the id column: identical id values (dedup/join key only, never used
positionally — verified no downstream .iloc/.loc/.index dependence), no O(E)
copy.

Consistent ~10-25ms lower per raw hop/chain traversal @500k/2M pandas; the
Cypher RETURN traversal is still dominated by binding materialization
(tracked #887). No user-frame contamination; hop + chain + gfql suites green
(152 + 3014 passed; 1 pre-existing umap env failure unrelated).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
@lmeyerov @claude
feat(gfql/polars): native rows(binding_ops) for fixed-length connecte…
ad4c53f 
…d patterns (#1709)

Native polars bindings-row tables — the rows(binding_ops=...) op emitted by
Cypher multi-alias lowering. Unblocks traversal-shaped Cypher on polars:
graph-benchmark q1/q2 (top-k in-degree), q8/q9 (fixed 2-hop counts),
multi-alias property projections, cross-alias grouped aggregates.

- binding_rows_polars: seed filter -> per-edge orient (fwd/rev/undirected,
  no-dedupe concat) + inner join -> node-filter semi join -> per-alias
  left-join assembly (alias.{col} node props, edge_alias.{col} payload,
  bare alias id cols). Same meaningful schema as pandas; pandas' internal
  join-residue columns intentionally not replicated.
- select_extend_polars: native with_(extend=True) (bindings-path aggregate
  lowering emits it; previously NIE'd).
- group_by key_prefixes guard: whole-row bindings grouping would have been
  silently dropped by the native dispatch once rows() stopped NIE-ing ->
  now declines honestly (latent wrong-answer trap).
- DECLINES (None -> honest NIE, NO-CHEATING): var-length/multihop edges,
  shortestPath scalar bindings, node query=/edge query/endpoint-match
  params, hop labels, HAS_-label disambiguation, seeded re-entry contexts,
  node-cartesian mode, alias_endpoints variant, join-key dtype divergence.

Conformance harness: _round_floats now renders non-bool numeric (incl
clean-coercible object) columns as float64 on both sides — pandas' hop
internals upcast int64->float64 on the 2nd alias (engine artifact; polars
faithfully keeps Int64), so the astype(str) gate failed on '20.0' vs '20'
for equal values. Same non-semantic noise class as the existing
summation-order rounding and null-repr normalization; genuine value
differences still fail.

Tests: +20 (parity incl q1/q8/q9 shapes, NIE assertions, raw-table schema,
path multiplicity, undirected self-loop); DEFERRED->supported moves in the
row-pipeline + conformance corpora. gfql suite 3036 passed; compute 4323
passed (9 pre-existing umap/dask env failures).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
@lmeyerov @claude
docs(changelog): #1709 polars binding_ops MVP entry
733f209 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
@lmeyerov @claude
feat(gfql/polars): bounded directed var-length binding rows (#1709, g…
6398d86 
…raph-bench q3)

Extends the native polars rows(binding_ops) builder (c9b178ad) to bounded
directed variable-length segments (-[*1..k]->, -[:TYPE*i..k]->, exactly-k):
iterative pair joins hop-by-hop, one row per distinct edge sequence (Cypher
path multiplicity; pairs not deduped so parallel edges multiply per hop —
matching pandas _gfql_multihop_binding_rows), zero-hop rows for min 0, same
min/max defaults as pandas (bare hops=k means exactly-k).

Unblocks the graph-benchmark q3 shape on polars:
  MATCH (a:Person)-[:FOLLOWS*1..2]->(b:Person) WHERE ... RETURN avg(b.age)
verified parity vs pandas (values + count) on typed-label replicas.

Still honestly deferred (NIE): unbounded [*] (needs fixed-point + termination
error), undirected var-length (immediate-backtrack avoidance not ported),
aliased var-length edges (pandas rejects outright). Non-aggregate var-length
projections route via the traversal chain (separate pre-existing Phase-1
single-hop limit) and keep their honest NIE.

Tests: +5 parity (q3 shape, exactly-k, typed, var+fixed compound, seeded avg),
DEFERRED repointed at unbounded/undirected. gfql suite 3042 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
@lmeyerov @claude
docs(changelog): fold var-length binding rows into the #1709 entry
56b44a8 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
@lmeyerov @claude
perf(gfql): projection-pushdown for binding tables — skip unused prop…
f0e8a21 
…erty joins (#1711)

The Cypher lowering now computes which node aliases have their PROPERTIES
referenced downstream and threads them as rows(binding_ops, attach_prop_aliases=...);
both binding builders (pandas _gfql_connected_bindings_row_frame_from_state and
polars binding_rows_polars) skip the O(N) property left-join for aliases not
listed — their free bare id column suffices. count(*) attaches zero property
columns; count(a) needs only a's bare id; RETURN b.id attaches only b.

Conservative gate (attach-all default when unsure): only computed for the simple
single-MATCH shape with no WITH stages, no WHERE anywhere, no repeated node alias
(bound-identity `n.__gfql_node_id__` refs a RETURN-text walk can't see, #1490),
and no collect() (carry/reentry hidden columns, #1413). The referenced set itself
is exact (_expr_match_alias_usage non-aggregate refs = property/whole-entity uses).

Primary win: unblocks the polars-gpu lazy-build (a large unpruned property-attached
intermediate made GPU q8 regress 19×; pruning → the join-chain the de-risk probe
clocked at ~12ms on GPU). Also trims polars-cpu q8 (244→185ms). pandas 2-hop is
dominated by the state-build merges, not the property attach, so it's ~unchanged
there (its bottleneck is separate).

Tests: +2 (pushdown skips unused props on both engines; end-to-end parity).
gfql 3042 + compute 4331 passed (9 pre-existing umap/dask env failures).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
@lmeyerov @claude
perf(gfql/polars-gpu): lazy binding build — collect once on target (#…
799b4bf 
…1709/#1711)

binding_rows_polars now builds ONE deferred pl.LazyFrame and collects once on the
active target (CPU / GPU via collect()). Under engine='polars-gpu' the whole join
chain + property attach runs on cudf_polars in a single GPU collect. Paired with
#1711 projection-pushdown (which prunes the unused property joins that otherwise
bloat the GPU intermediate), this delivers the polars-gpu OLAP win WITHOUT the
regression the lazy-build-alone caused.

Column access uses .collect_schema().names() (schema-only, no data). The
var-length loop drops the eager .height early-break (empty lazy joins are
identical; the break was an optimization). NO-CHEATING preserved: a GPU-incapable
plan node makes collect() raise NotImplementedError (honest NIE), never a silent
CPU fallback.

Validated on dgx (safe_run, all values correct) @100k nodes/1M edges:
  q8 2-hop count polars-gpu: 4294ms (lazy-alone regression) -> 54ms (79x); now the
    FASTEST engine (vs polars-cpu 90, cudf 439, pandas 1122).
  q1 top-k polars-gpu 30ms (fastest). #1711 also cut all engines on q8
    (pandas 4386->1122, cudf 973->439, polars 244->90).
(One transient GPU-JIT spike at 50k — non-monotonic 9/311/54ms across sizes.)

gfql suite 3044 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
@lmeyerov @claude
test(gfql/cypher): lock #1712 subset WITH-carry wrong-answer (benchma…
38758f7 
…rk q5/q6/q7 shape)

The benchmark shortcuts bypassed the GFQL engine, so the multi-part
MATCH...WHERE...WITH <subset> MATCH (alias)... shape (graph-benchmark
q5/q6/q7) had NO real-engine correctness test — which is how the silent
wrong answer in #1712 went uncaught. Adds the missing xfail(strict)-locked
correctness test: a >1-node filtered subset carried via WITH must restrict
the re-matched alias (numPersons=2, not 3), with and without collect.
Flips to pass when #1712 lands; strict xfail forces re-enabling then.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
@lmeyerov @claude
fix(gfql/cypher): connected multi-pattern + WITH-carry drop the share…
3e475bc 
…d-alias filter (#1712 silent wrong)

Two manifestations of the same silent wrong answer — a connected pattern
sharing a node alias failed to intersect the two patterns' constraints,
returning unfiltered counts (graph-benchmark q2/q5/q6/q7 shape). Both fixed:

1. Comma-form `MATCH (p)-[R1]->(i),(p)-[R2]->(c) WHERE i.k='x'`: the
   connected-match-join only applied a WHERE in the `expr_tree` form and
   SILENTLY DROPPED the structured `predicates` form (a plain `WHERE i.k='x'`).
   `_where_clause_expr_text` already renders both forms; relax the gate and
   NIE honestly if unrenderable rather than drop.

2. WITH-form `MATCH (p)-[R1]->(i) WHERE i.k='x' WITH p MATCH (p)-[R2]->(c)`:
   the bounded-reentry seed (the prefix-filtered `p`) was computed and passed
   as chain start_nodes, but never wired to `_gfql_start_nodes` for an all-call
   binding-ops chain (only the traversal->suffix-call boundary path set it), so
   the binding builder re-matched `p` from the whole graph. Wire it in
   `_execute_compiled_query_chain_non_union` when the chain is a
   rows(binding_ops) pipeline.

Both correct on pandas and cuDF now (numPersons=2, not 3). This unblocks the
benchmark q5/q6/q7 (expressible as the comma-form with tolower).

Tests: the previously xfail(strict)-locked subset-carry test now passes
(WITH + WITH-collect forms) + a comma-form intersection test. gfql 3047 +
compute 4334 passed (9 pre-existing umap/dask env failures).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
@lmeyerov @claude
fix(gfql/cypher): multi-source grouped aggregate projection (#1273, g…
0f4870a 
…raph-bench q3/q4)

`RETURN c.city, avg(p.age)` (group key on one alias, aggregate over another)
NIE'd with the misleading "one MATCH source alias at a time". Now routes to
the bindings-row table (which materializes every alias, one row per matched
path — a standard GROUP BY). Extends the #1708 force-bindings gate from
bare-alias count to any CLEAN grouped aggregate func(<alias>.<prop>)
(avg/sum/min/max/count) over a non-active node alias.

Guarded for soundness: only CLEAN agg/non-agg separation is admitted — an
aggregate nested inside a larger expression (`me.age + count(you.age)`, or
`age + count(...)` in ORDER BY) keeps the conservative fail-fast (the
rejects-unsound-multi-source-overlap contract), detected structurally via
_expr_has_aggregate / _is_pure_aggregate_call over RETURN + top-level ORDER BY.

Also fixes a latent #1711 bug this exposed: _binding_prop_alias_set dropped an
alias referenced only via a property inside an aggregate (`avg(p.age)` needs
p's join) — it now includes property-access aliases from inside aggregates
(_expr_property_access_node_aliases), and reads the top-level query.order_by
(ReturnClause has no order_by attr).

avg/sum/count(property) correct on pandas + cuDF (covers q3 avg, q4 count).
min/max still NIE honestly (not multiplicity-sensitive, so they don't reach
the force-bindings gate — a separate, non-benchmark-blocking gap). +3 tests.
gfql 3050 + compute 4337 passed (9 pre-existing umap/dask env failures).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
@lmeyerov @claude
bench(gfql): real-GFQL q1-q9 runner — no shortcuts, no untimed precom…
32f4e13 
…pute (#1710)

Every query runs g.gfql(<canonical cypher>) on the GFQL schema (node_type/rel
inline maps, toLower inline+timed); q2 orchestrated as two real GFQL calls;
q5/q6/q7 as comma-form. Times pandas/cudf/polars/polars-gpu, values checked vs
pandas. Verified on tiny data: pandas all q1-q9+q2; polars matches on
q1/q2/q3/q4/q8/q9 (q5/q6/q7 NIE — comma-form connected-join not polars-native).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
lmeyerov added 10 commits July 8, 2026 21:06
@lmeyerov
perf(gfql): optimize graph-benchmark OLAP lowering
b2f09c2 
Reduce real GFQL q1-q9 OLAP overhead with predicate pushdown, Polars-native binding fast paths, count-without-materialization paths, reusable graph-local caches, and compiled string-query plan caching.

Add known-answer and cache-policy coverage for the optimized GFQL/Cypher paths and keep the benchmark runner lint-clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
@lmeyerov
test(gfql): cover OLAP lowering helper paths
f1002be
@lmeyerov
test(gfql): cover conservative OLAP lowering branches
3b66bee
@lmeyerov
ci(gfql): expose coverage audit baseline failures
87cac9b
@lmeyerov
ci(gfql): exclude polars fast paths from pandas coverage
a213da1
@lmeyerov
test(gfql): cover pandas aggregate fast-path branches
2c901e9
@lmeyerov
ci(gfql): exclude singleton polars count branch
23fa498
@lmeyerov
fix(gfql): route polars single-node binding rows
36dfc34
@lmeyerov
test(gfql): cover polars binding rows in CI
7ce116a
@lmeyerov
ci(gfql): refresh cypher surface guard after residual stack
321fe95
@lmeyerov lmeyerov force-pushed the perf/gfql-olap-real-q1q9 branch from b09397f to 321fe95 Compare July 9, 2026 04:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@lmeyerov