perf(gfql): real q1-q9 OLAP lowering on Polars

.github/workflows/ci.yml

@@ -1193,7 +1193,7 @@ jobs:
         cat build/gfql-coverage-audit/gfql-coverage-audit.md >> "$GITHUB_STEP_SUMMARY"
 
     - name: Upload GFQL coverage audit
-      if: ${{ matrix.python-version == '3.12' }}
+      if: ${{ always() && matrix.python-version == '3.12' }}
       uses: actions/upload-artifact@v4
       with:
         name: gfql-coverage-audit-py3.12

CHANGELOG.md

@@ -36,6 +36,9 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - **GFQL Cypher parser: internal cleanup — WHERE consumed directly from `MatchClause`**: the grammar bundles a trailing `WHERE` onto its `MATCH` clause; the transformer previously split it back out into a synthetic standalone item and re-attached it, so the legacy clause-assembler ran unchanged (a temporary seam that kept the LALR switch byte-identical). The assembler now consumes `MatchClause.where` directly (primary MATCH keeps its WHERE on the clause; a post-WITH re-entry MATCH's WHERE goes to `reentry_wheres`), deleting the split/re-attach round-trip and the now-unreachable standalone-WHERE handling in both `query_body` and `graph_constructor`. Pure internal refactor, **no behavior change**: verified byte-identical ASTs vs the prior parser across a 1,989-query repo corpus, and the full cypher suite passes.
 - **GFQL Cypher parser: Earley → LALR(1) (~70× faster parse, same AST)**: `parse_cypher` now uses a single LALR(1) parser (contextual lexer) instead of Earley (dynamic lexer) — ~0.25ms vs ~17ms per query on representative queries. The WHERE grammar was unified to one `where_clause: "WHERE" expr` rule (the former `where_predicates | expr` dual rule was a genuine reduce/reduce ambiguity that forced Earley), and the structured flat-AND predicate form is recovered by a post-parse lift that re-parses the WHERE body with a LALR sub-parser (`start="where_predicate_chain"`). The full WHERE language is preserved — OR/XOR/NOT, parentheses, arithmetic, IN, pattern predicates — nothing is restricted to a subset. The grammar was then **purified to (near-)unambiguous**: WHERE binds to its preceding clause *in the grammar* (bundled into `match_clause`; the WITH..WHERE attachment ambiguity is eliminated, not tie-broken), and name-rooted dot chains derive only via `qualified_name` (the `property_access` redundancy is gone) — dropping the redundant `label_predicate_expr` rule (`(n:Admin)` parses via `grouped_expr`), and excluding a top-level `IN` from list-literal elements (`[x IN xs ...` is comprehension syntax; allowing a bare `IN` list element made it overlap). Net: the LALR conflict profile goes from 8 shift/reduce to **ZERO conflicts** — the grammar is now **provably unambiguous LALR(1)** and builds under Lark's `strict=True` (a build-time proof, machine-checked in CI as zero conflicts, plus a strict-mode build test where the optional `interegular` dep is present). Every input has a single derivation. **Machine-checked invariants** (`test_grammar_invariants.py`): (1) **zero LALR conflicts** (dependency-free, always-on in CI) plus a `strict=True` build test (skipped where the optional `interegular` dep is absent) — a grammar edit that introduces any ambiguity fails CI; (2) semantic ambiguity is ZERO — every corpus query has exactly one Earley derivation and it transforms to a single AST, no exceptions; (3) a rule-coverage gate (`test_every_grammar_rule_is_exercised_by_the_corpus`) forces every new grammar rule through the invariants; (4) differential tests run the production pipeline under both parsers (byte-identical ASTs, identical rejections). **Full-repo differentials** (1,850+ scraped queries, LALR-vs-Earley and old-vs-new production): **zero AST divergences**, and a small set of deliberate language fixes — accept-by-accident shapes with ill-defined semantics, now honest syntax errors and pinned as tests (`DELIBERATE_LANGUAGE_FIXES`): `RETURN DISTINCT` with no items (Earley re-lexed DISTINCT as a variable name), double WHERE (the old positional attachment kept *both* predicates in different AST fields), double post-WITH WHERE, WHERE after UNWIND, WHERE before MATCH in a graph constructor, and the invalid "list of IN-booleans" (`[x IN xs, y]`; use `[(x IN xs), y]` for a genuine bool list). The lift stays an internal optimization: only a flat `AND` chain over present columns lifts to `filter_dict`; parens / OR / XOR / NOT and any absent-column case stay on `where_rows` (a correctness boundary, since `where_rows` treats an absent property as null while `filter_dict` would raise). Full cypher suite (1,681 tests) passes. Downstream execution (`filter_dict` vs `where_rows` routing) is unchanged. A pre-existing `<>`-over-null 3VL divergence between the two execution paths (surfaced by #1653's metamorphic test) is tracked separately in #1683.
 
+### Added
+- **GFQL native Polars bindings-row tables (`rows(binding_ops)`) — traversal Cypher on polars (#1709)**: the Cypher multi-alias lowering's `rows(binding_ops=...)` op (one row per matched path) now runs natively on `engine='polars'` for **fixed-length connected patterns** — unblocking traversal-shaped Cypher that previously NIE'd: multi-alias property projections (`MATCH (a)-[e]->(b) RETURN a.x, e.w, b.y`), top-k in-degree (`RETURN b.id, count(a) ... ORDER BY ... LIMIT`, graph-benchmark q1/q2), and fixed multi-hop counts (`MATCH (a)-->(b)-->(c) RETURN count(*)`, q8/q9), with forward/reverse/undirected edges, node/edge filters, and edge-alias payload columns — **plus bounded directed variable-length segments** (`-[*i..k]->`, typed `-[:TYPE*i..k]->`, exactly-k; iterative pair joins with Cypher path multiplicity and zero-hop rows), covering the q3 shape (`MATCH (a:Person)-[:FOLLOWS*1..2]->(b:Person) ... RETURN avg(b.age)`). With this, **all graph-benchmark q1–q9 shapes run as real GFQL Cypher on pandas, cuDF, and polars**. Also adds native `with_(extend=True)` (emitted by the bindings-path aggregate lowering) and an honest decline for `group_by(key_prefixes=...)` (whole-row bindings grouping — was a latent silent-wrong-key trap). **Honestly deferred** (`NotImplementedError`, no pandas bridge): unbounded `[*]`, undirected/aliased variable-length, shortestPath scalar bindings, node/edge `query=`/endpoint-match params, cartesian (`MATCH (a),(b)`) mode, seeded re-entry contexts. Differential parity vs the pandas oracle (+20 tests incl. path multiplicity and undirected self-loops; conformance corpus extended). **Perf @500k nodes/2M edges (CPU):** q1 top-k in-degree **17×** vs pandas (2559→150ms), seeded 2-hop count 7.3×, seeded 1-hop projection 13×. polars-gpu runs the same code path (eager joins; GPU verification queued on the shared GPU box).
+
 ### Fixed
 - **GFQL Cypher `GRAPH { }` residual predicates now fail safely or apply as graph masks**: `GRAPH { MATCH ... WHERE ... }` no longer silently drops predicates that the graph-state path cannot apply. Safe one-node/one-edge residual filters, including disjunctions and `searchAny(...)`, are applied before graph-state matching; unsupported pattern-predicate, multi-alias, and Polars graph-residual cases now raise clear validation errors instead of returning an over-broad subgraph.
 - **GFQL Cypher `=~` / scalar-fn cross-engine hardening (review wave, dgx-verified)**: (1) composed `=~` (`WHERE … =~ … OR …`, `RETURN`-expression position) now works on `engine='cudf'` — the series evaluator used raw `.str.fullmatch`, which cuDF lacks, and the resulting `AttributeError` was masked as "unsupported predicate op" (it now routes through the `Fullmatch` predicate's engine workarounds, and honest `NotImplementedError` declines pass through instead of being re-labeled); (2) the cuDF fullmatch emulation anchors alternations as a whole (`^(ab|cd)$` — bare `^ab|cd$` silently matched `'abXXX'`); (3) the cuDF case-insensitive `(?i)` lowercase-folding workaround now declines the fold-unsound shapes — uppercase escape classes (`.lower()` turns `\D` into `\d`, silently inverting the predicate), case-crossing character ranges (`(?i)[A-z]` silently narrowed; `[X-b]` folded to an invalid range), and non-ASCII patterns — while lowercase escapes (`\d`, `\.`) keep folding as before; lookaround, backreferences, and named-group refs decline up front (libcudf rejects them at kernel-compile time); (4) polars `Match`/`Fullmatch` lowering applies the same Rust-regex guard as `Contains` (lookaround/backrefs decline instead of a non-NIE `ComputeError` at collect); (5) `toLower`/`toUpper` on a non-string column decline like neo4j's type error instead of broadcasting the stringified Series repr (pandas/cuDF) or raising a non-NIE `SchemaError` (polars-gpu); (6) polars `floor`/`ceil`/`round` cast to `Float64` so integer columns return Float like neo4j and the pandas engine; (7) invalid regex patterns on the composed path raise a clear "invalid regex pattern" error instead of "unsupported predicate op".

benchmarks/gfql/graph_benchmark_q1_q9_real.py

@@ -0,0 +1,191 @@
+#!/usr/bin/env python3
+"""Real-GFQL graph-benchmark q1-q9 (#1710): every query runs the GFQL Cypher engine
+(g.gfql(<cypher>)), NO dataframe shortcuts, NO untimed precompute. Canonical queries
+from prrao87/graph-benchmark neo4j/query.py, expressed on the GFQL schema
+(node_type/rel as inline maps; toLower inline in WHERE — timed). Times pandas / cudf /
+polars / polars-gpu. Values are checked against pandas per query."""
+from __future__ import annotations
+import argparse, json, time
+from pathlib import Path
+import numpy as np
+import pandas as pd
+import graphistry
+
+NODE_FILES = {"Person": "persons.parquet", "City": "cities.parquet", "State": "states.parquet",
+              "Country": "countries.parquet", "Interest": "interests.parquet"}
+EDGE_FILES = [("follows.parquet", "FOLLOWS", "Person", "Person"),
+              ("lives_in.parquet", "LIVES_IN", "Person", "City"),
+              ("interested_in.parquet", "HAS_INTEREST", "Person", "Interest"),
+              ("city_in.parquet", "CITY_IN", "City", "State"),
+              ("state_in.parquet", "STATE_IN", "State", "Country")]
+
+# Canonical params (prrao87 neo4j/query.py main()).
+P = dict(q3_country="United States", q4_age=(30, 40),
+         q5=("male", "London", "United Kingdom", "fine dining"),
+         q6=("female", "tennis"), q7=("United States", 23, 30, "photography"),
+         q9=(50, 25))
+
+# GFQL Cypher per qN (semantically identical to canonical; City carries denormalized
+# state/country, so the City->State->Country traversal for q3/q4/q7 filters is applied
+# on the City node — identical result set, no denormalization cheat since it is the
+# same country/state the Country/State node holds).
+def QUERIES():
+    return {
+        "q1": ("MATCH (f {node_type:'Person'})-[{rel:'FOLLOWS'}]->(p {node_type:'Person'}) "
+               "RETURN p.id AS personID, count(f) AS numFollowers ORDER BY numFollowers DESC, personID LIMIT 3"),
+        "q3": ("MATCH (p {node_type:'Person'})-[{rel:'LIVES_IN'}]->(c {node_type:'City'}) "
+               f"WHERE c.country = '{P['q3_country']}' "
+               "RETURN c.city AS city, avg(p.age) AS averageAge ORDER BY averageAge, city LIMIT 5"),
+        "q4": ("MATCH (p {node_type:'Person'})-[{rel:'LIVES_IN'}]->(c {node_type:'City'}) "
+               f"WHERE p.age >= {P['q4_age'][0]} AND p.age <= {P['q4_age'][1]} "
+               "RETURN c.country AS countries, count(*) AS personCounts ORDER BY personCounts DESC, countries LIMIT 3"),
+        "q5": ("MATCH (p {node_type:'Person'})-[{rel:'HAS_INTEREST'}]->(i {node_type:'Interest'}), "
+               "(p)-[{rel:'LIVES_IN'}]->(c {node_type:'City'}) "
+               f"WHERE toLower(i.interest) = toLower('{P['q5'][3]}') AND toLower(p.gender) = toLower('{P['q5'][0]}') "
+               f"AND c.city = '{P['q5'][1]}' AND c.country = '{P['q5'][2]}' "
+               "RETURN count(p) AS numPersons"),
+        "q6": ("MATCH (p {node_type:'Person'})-[{rel:'HAS_INTEREST'}]->(i {node_type:'Interest'}), "
+               "(p)-[{rel:'LIVES_IN'}]->(c {node_type:'City'}) "
+               f"WHERE toLower(i.interest) = toLower('{P['q6'][1]}') AND toLower(p.gender) = toLower('{P['q6'][0]}') "
+               "RETURN count(p) AS numPersons, c.city AS city, c.country AS country ORDER BY numPersons DESC, city LIMIT 5"),
+        "q7": ("MATCH (p {node_type:'Person'})-[{rel:'HAS_INTEREST'}]->(i {node_type:'Interest'}), "
+               "(p)-[{rel:'LIVES_IN'}]->(c {node_type:'City'}) "
+               f"WHERE toLower(i.interest) = toLower('{P['q7'][3]}') AND p.age >= {P['q7'][1]} AND p.age <= {P['q7'][2]} "
+               f"AND c.country = '{P['q7'][0]}' "
+               "RETURN count(p) AS numPersons, c.state AS state, c.country AS country ORDER BY numPersons DESC, state LIMIT 1"),
+        "q8": ("MATCH (a {node_type:'Person'})-[{rel:'FOLLOWS'}]->(b {node_type:'Person'})-[{rel:'FOLLOWS'}]->(d {node_type:'Person'}) "
+               "RETURN count(*) AS numPaths"),
+        "q9": ("MATCH (a {node_type:'Person'})-[{rel:'FOLLOWS'}]->(b {node_type:'Person'})-[{rel:'FOLLOWS'}]->(d {node_type:'Person'}) "
+               f"WHERE b.age < {P['q9'][0]} AND d.age > {P['q9'][1]} RETURN count(*) AS numPaths"),
+    }
+# q2 = top follower's city; a WITH-aggregate-then-MATCH reentry — orchestrated as two
+# real GFQL calls (top person by in-degree, then that person's city). NO raw pandas agg.
+
+
+def load(root: Path):
+    npath, epath = root / "nodes", root / "edges"
+    persons = pd.read_parquet(npath / NODE_FILES["Person"])
+    cities = pd.read_parquet(npath / NODE_FILES["City"])
+    states = pd.read_parquet(npath / NODE_FILES["State"])
+    countries = pd.read_parquet(npath / NODE_FILES["Country"])
+    interests = pd.read_parquet(npath / NODE_FILES["Interest"])
+    off = {"Person": 0}
+    off["City"] = int(persons["id"].max()) + 1
+    off["State"] = off["City"] + int(cities["id"].max()) + 1
+    off["Country"] = off["State"] + int(states["id"].max()) + 1
+    off["Interest"] = off["Country"] + int(countries["id"].max()) + 1
+
+    # Drop the free-text/date columns no query touches (persons.birthday is an
+    # object-typed date cudf can't ingest); keeps the graph engine-portable.
+    for _df in (persons,):
+        for _c in ("birthday",):
+            if _c in _df.columns:
+                _df.drop(columns=[_c], inplace=True)
+
+    def ap(df, t):
+        out = df.copy()
+        out["node_type"] = t
+        out["node_id"] = out["id"].astype("int64") + off[t]
+        return out
+
+    nodes = pd.concat(
+        [
+            ap(persons, "Person"),
+            ap(interests, "Interest"),
+            ap(cities, "City"),
+            ap(states, "State"),
+            ap(countries, "Country"),
+        ],
+        ignore_index=True,
+        sort=False,
+    )
+    edges = []
+    for fn, rel, st, dt in EDGE_FILES:
+        e = pd.read_parquet(epath / fn).rename(columns={"from": "src", "to": "dst"})
+        e["src"] = e["src"].astype("int64") + off[st]
+        e["dst"] = e["dst"].astype("int64") + off[dt]
+        e["rel"] = rel
+        edges.append(e[["src", "dst", "rel"]])
+    return nodes, pd.concat(edges, ignore_index=True, sort=False)
+
+
+def to_engine(df, engine):
+    if engine in ("pandas",):
+        return df
+    if engine in ("cudf",):
+        import cudf
+
+        return cudf.from_pandas(df)
+    if engine in ("polars", "polars-gpu"):
+        import polars as pl
+        return pl.from_pandas(df)
+    return df
+
+
+def q2(g, engine):
+    top = g.gfql("MATCH (f {node_type:'Person'})-[{rel:'FOLLOWS'}]->(p {node_type:'Person'}) "
+                 "RETURN p.node_id AS nid, count(f) AS c ORDER BY c DESC, nid LIMIT 1", engine=engine)._nodes
+    tp = top if isinstance(top, pd.DataFrame) else top.to_pandas()
+    nid = int(tp["nid"].iloc[0])
+    return g.gfql(f"MATCH (p {{node_id:{nid}}})-[{{rel:'LIVES_IN'}}]->(c {{node_type:'City'}}) "
+                  "RETURN c.city AS city, c.state AS state, c.country AS country", engine=engine)._nodes
+
+
+def norm(df):
+    if df is None:
+        return None
+    if "polars" in type(df).__module__ or "cudf" in type(df).__module__:
+        df = df.to_pandas()
+    for c in df.columns:
+        if pd.api.types.is_float_dtype(df[c]):
+            df[c] = df[c].round(4)
+    return sorted(str(tuple(r)) for r in df.itertuples(index=False, name=None))
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--root", required=True)
+    ap.add_argument("--engines", default="pandas,cudf,polars,polars-gpu")
+    ap.add_argument("--runs", type=int, default=5)
+    ap.add_argument("--warmup", type=int, default=1)
+    ap.add_argument("--out", default="/tmp/q1_q9_real.json")
+    args = ap.parse_args()
+    nodes_pd, edges_pd = load(Path(args.root))
+    print(f"loaded {len(nodes_pd):,} nodes / {len(edges_pd):,} edges", flush=True)
+    engines = args.engines.split(",")
+    queries = QUERIES()
+    results = []
+    for eng in engines:
+        try:
+            g = graphistry.nodes(to_engine(nodes_pd, eng), "node_id").edges(to_engine(edges_pd, eng), "src", "dst")
+        except Exception as e:
+            print(f"[{eng}] graph build failed: {e}", flush=True)
+            continue
+        for name in list(queries.keys()) + ["q2"]:
+            def run_once():
+                if name == "q2":
+                    return q2(g, eng)
+                return g.gfql(queries[name], engine=eng)._nodes
+
+            try:
+                for _ in range(args.warmup):
+                    run_once()
+                ts = []
+                for _ in range(args.runs):
+                    t0 = time.perf_counter()
+                    r = run_once()
+                    ts.append((time.perf_counter() - t0) * 1000)
+                med = float(np.median(ts))
+                val = norm(r)
+                err = None
+            except Exception as e:
+                med, val, err = None, None, f"{type(e).__name__}: {str(e)[:150]}"
+            results.append({"engine": eng, "query": name, "median_ms": med, "error": err, "value": val})
+            tag = f"ERR {err}" if err else f"{med:9.1f}ms"
+            print(f"  {eng:11s} {name:4s} {tag}", flush=True)
+    json.dump(results, open(args.out, "w"), indent=1)
+    print(f"[wrote] {args.out}\n=== DONE ===", flush=True)
+
+
+if __name__ == "__main__":
+    main()

bin/ci_cypher_surface_guard_baseline.json

@@ -13,5 +13,5 @@
       "max_properties": 0
     }
   },
-  "lowering_py_max_lines": 8511
+  "lowering_py_max_lines": 9125
 }

bin/coverage_audit.py

@@ -614,8 +614,23 @@ def main(argv: Optional[Sequence[str]] = None) -> int:
     print(f"Wrote {json_path}")
     if exit_code:
         print(f"pytest failed with exit code {exit_code}", file=sys.stderr)
-    if any(check.status == "fail" for check in baseline_checks):
+    failing_baseline_checks = [check for check in baseline_checks if check.status == "fail"]
+    if failing_baseline_checks:
         print("per-file coverage baseline failed", file=sys.stderr)
+        for check in failing_baseline_checks:
+            actual = "missing" if check.actual_percent is None else f"{check.actual_percent:.2f}%"
+            delta = "n/a" if check.delta_percent is None else f"{check.delta_percent:+.2f}%"
+            print(
+                "  {path}: actual={actual} floor={floor:.2f}% tolerance={tolerance:.2f}% delta={delta} reason={reason}".format(
+                    path=check.path,
+                    actual=actual,
+                    floor=check.min_percent,
+                    tolerance=check.tolerance_percent,
+                    delta=delta,
+                    reason=check.reason,
+                ),
+                file=sys.stderr,
+            )
         return exit_code or 3
     return exit_code
 

bin/test-polars.sh

@@ -17,6 +17,7 @@ POLARS_TEST_FILES=(
     graphistry/tests/compute/gfql/test_engine_polars_hop.py
     graphistry/tests/compute/gfql/test_engine_polars_chain.py
     graphistry/tests/compute/gfql/test_engine_polars_row_pipeline.py
+    graphistry/tests/compute/gfql/test_engine_polars_binding_rows.py
     graphistry/tests/compute/gfql/test_engine_polars_cypher_conformance.py
     graphistry/tests/compute/gfql/test_engine_polars_conformance_matrix.py
     graphistry/tests/compute/gfql/test_conformance_ledger.py