docs(gfql): engine-selection guide (pandas/polars/cuDF/polars-gpu) + motivating comparison

CHANGELOG.md

@@ -8,6 +8,9 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 ## [Development]
 <!-- Do Not Erase This Section - Used for tracking unreleased changes -->
 
+### Documentation
+- **GFQL engine-selection docs (pandas / polars / cuDF / polars-gpu)**: New :doc:`Choosing a GFQL Engine <gfql/engines>` page — a numbers-first, persona-tested guide to the four interchangeable engines. Adds the one-keyword `engine='polars'` speedup (up to ~38× over pandas on real graphs, no GPU), a motivating warm-median comparison table on real public graphs (LiveJournal 35M / Orkut 117M), a decision matrix (workload shape × size × hardware → engine, with the measured ~10K-edge CPU crossover, GPU-work-bound rule, polars-gpu memory-pressure caveat, and GPU-or-error contract), a cuDF-vs-polars-gpu disambiguation (eager-op vs fused-lazy; cuDF is not deprecated), an honest "when *not* to use Polars" section, the differential-parity guarantee, and a methodology + reproducer-script disclosure. Rewrote the top of `gfql/performance.rst` to lead with the engine comparison (de-marketed the prose), wired the new page into the GFQL toctree + recommended paths, and added Polars/polars-gpu to the engine examples in `gfql/quick.rst` and `gfql/about.rst` (previously only pandas/cuDF were documented). Driven by 4-persona doc user-testing (pandas DS, RAPIDS user, perf engineer, skeptical evaluator).
+
 ### Added
 - **GFQL viz-filter-pipeline acceptance suite + regression benchmark (viz-filter L3)**: `test_viz_pipeline_conformance.py` — curated full-panel pipelines (node+edge filters, exclusion-dominates composition, `(pred OR IS NULL)` keep-null leaves, both EXISTS prune-isolated flavors, searchAny composition, deterministic paging), graph-state prune shapes with exact node+edge pins, a 40-seed panel-state fuzzer with an independent plain-pandas second oracle, and a case/regex/unicode trick matrix (ß/İ full-case-mapping pins, metachar literal-vs-regex, null cells, Categorical) — all parity-or-NIE across pandas/cuDF/polars/polars-gpu. `benchmarks/gfql/viz_filter_pipeline.py` — six streamgl-viz panel scenarios (filters, keep-self GRAPH prune, EXISTS prune, node/edge search, combined) at 100K/1M/10M with native-frame-per-engine fairness, an NIE-tolerant matrix, and JSON receipts (first receipt: 100k × 4 engines, everything within the ~350ms interactive reference except pandas combined). Documented findings from the first runs: the same-path WHERE route dedupes parallel edges (diverges from the panel algebra's edge multiplicity — pinned + tracked), and edge-alias searchAny declines on polars (tracked).
 - **GFQL Cypher `searchAny(entity, term[, opts])` cross-column search predicate + `g.search_nodes()`/`g.search_edges()` (viz-filter L2, native on all four engines)**: True where ANY of the entity's columns matches the term — the streamgl-viz inspector's table-search semantics as a composable WHERE predicate: OR across columns; case-insensitive substring by DEFAULT (case-folded, never regex — the common call avoids every engine regex limit); regex opt-in obeying the same per-engine decline rules as `=~`; dtype gate AS SEMANTICS (string columns always; integer columns iff the term is a numeric literal, per the inspector's `/^[0-9.-]+$/` gate; floats/dates/booleans reachable via the explicit `columns:` list). Options map `{caseSensitive, regex, columns}` is strict-validated (unknown keys error listing the valid ones); unbound aliases and missing explicit columns error clearly; null cells never match. Lowered like the pattern-predicate markers (a `search_any` row op + fresh marker column), so it composes through AND/OR/NOT and different node/edge terms coexist in one pipeline. Per-column matching reuses the parity-hardened `Contains` predicate on pandas/cuDF and a lowercase-fold/any_horizontal lowering on polars; oracle-pinned + 4-engine parity-or-NIE conformance cases. Python twins `g.search_nodes(term, columns=, case_sensitive=, regex=)` / `g.search_edges(...)` filter their own table and return a Plottable (polars-frame twins decline honestly for now — use the cypher op). Honest declines (NIE, use engine='pandas'): edge-alias searchAny on polars, and explicit columns beyond string/int/bool dtypes on polars AND cuDF (incl. floats: repr diverges across engines — dgx-probed).

benchmarks/gfql/index_crossover_bench.py

@@ -0,0 +1,64 @@
+#!/usr/bin/env python3
+"""Small-N pandas-vs-polars CROSSOVER bench (CPU). Answers "where does polars start
+beating pandas?" per workload SHAPE, on a real graph subsampled to N edges.
+
+The crossover is shape-dependent: row-pipeline shapes (filter / WHERE+ORDER) cross over
+much earlier than traversal (chain orchestration is the residual small-N fixed cost).
+CPU only (the crossover question is pandas-CPU vs polars-CPU); no GPU needed.
+
+Env: PARQUET=/data/edges.parquet  EDGES=10000,100000,1000000  REPS=15  WARM=3  OUT=/tmp/x.jsonl
+"""
+from __future__ import annotations
+import json, os, statistics, time
+import numpy as np
+import pandas as pd
+import graphistry
+from graphistry.compute.ast import n, e_forward
+
+
+def med(fn, reps, warm):
+    for _ in range(warm):
+        fn()
+    ts = []
+    for _ in range(reps):
+        t = time.perf_counter(); fn(); ts.append((time.perf_counter() - t) * 1e3)
+    ts.sort()
+    return statistics.median(ts)
+
+
+def main():
+    edf_full = pd.read_parquet(os.environ["PARQUET"]).astype({"src": np.int64, "dst": np.int64})
+    sizes = [int(x) for x in os.environ.get("EDGES", "10000,100000,1000000").split(",")]
+    reps = int(os.environ.get("REPS", "15")); warm = int(os.environ.get("WARM", "3"))
+    outf = open(os.environ["OUT"], "a") if os.environ.get("OUT") else None
+    print(f"{'shape':10} {'edges':>9} {'pandas_ms':>10} {'polars_ms':>10} {'polars_speedup':>15}")
+    for E in sizes:
+        edf = edf_full.head(E).reset_index(drop=True)
+        nodes = np.unique(np.concatenate([edf["src"].values, edf["dst"].values]))
+        ndf = pd.DataFrame({"id": nodes, "val": (nodes % 100).astype(np.int64)})
+        g = graphistry.nodes(ndf, "id").edges(edf, "src", "dst")
+        seeds = nodes[: max(1, len(nodes) // 100)].tolist()  # ~1% frontier
+        shapes = {
+            "filter": lambda eng: g.gfql([n({"val": 50})], engine=eng),
+            "hop1": lambda eng: g.gfql([n({"id": seeds}), e_forward()], engine=eng),
+            "where_ord": lambda eng: g.gfql(
+                "MATCH (a) WHERE a.val > 50 RETURN a.id ORDER BY a.id LIMIT 100", engine=eng),
+        }
+        for name, fn in shapes.items():
+            try:
+                rp = fn("pandas"); rl = fn("polars")  # warm + sanity
+                pm = med(lambda: fn("pandas"), reps, warm)
+                lm = med(lambda: fn("polars"), reps, warm)
+                sp = pm / lm if lm else float("nan")
+                print(f"{name:10} {E:>9} {pm:>10.3f} {lm:>10.3f} {('polars '+format(sp,'.2f')+'x') if sp>=1 else ('PANDAS '+format(1/sp,'.2f')+'x'):>15}")
+                if outf:
+                    outf.write(json.dumps(dict(shape=name, edges=E, pandas_ms=pm, polars_ms=lm,
+                                               polars_speedup=sp)) + "\n"); outf.flush()
+            except Exception as ex:
+                print(f"{name:10} {E:>9}  FAILED {type(ex).__name__}: {ex}")
+    if outf:
+        outf.close()
+
+
+if __name__ == "__main__":
+    main()

docs/source/gfql/about.rst

@@ -27,7 +27,7 @@ GFQL fills a critical gap in the data community by providing an in-process, high
 
 **Key Benefits:**
 
-- **Dataframe-Native:** Works directly with Pandas, cuDF, and other dataframe libraries.
+- **Dataframe-Native:** Works directly with Pandas, Polars, cuDF, and other dataframe libraries.
 - **High Performance:** Optimized for both CPU and GPU execution.
 - **Ease of Use:** No need for external databases or new infrastructure.
 - **Interoperability:** Integrates with the Python data science ecosystem, including PyGraphistry for visualization.
@@ -372,21 +372,30 @@ GFQL is optimized for GPU acceleration using ``cudf`` and ``rapids``. When using
 - GFQL detects ``cudf`` dataframes and runs the query on the GPU.
 - Achieves significant performance improvements on large datasets.
 
-7. Forcing GPU Mode
-~~~~~~~~~~~~~~~~~~~~
+7. Selecting an Engine (CPU and GPU)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-You can explicitly set the engine to ensure GPU execution.
+You can explicitly set the execution engine. The same query returns identical
+results on every engine — see :doc:`Choosing an Engine <engines>`.
 
-**Example: Force GFQL to use GPU engine**
+**Example: CPU columnar speedup (no GPU)**
 
 ::
 
-    g_result = g_gpu.gfql([ ... ], engine='cudf')
+    g_result = g.gfql([ ... ], engine='polars')   # up to ~38x over pandas on real graphs
+
+**Example: Force GFQL to use a GPU engine**
+
+::
+
+    g_result = g_gpu.gfql([ ... ], engine='cudf')        # NVIDIA GPU, eager
+    g_result = g_gpu.gfql([ ... ], engine='polars-gpu')  # NVIDIA GPU, fused plan
 
 **Explanation:**
 
-- ``engine='cudf'`` forces the use of the GPU-accelerated engine.
-- Useful when you want to ensure the query runs on the GPU.
+- ``engine='polars'`` runs the columnar CPU engine — the biggest win without a GPU.
+- ``engine='cudf'`` / ``'polars-gpu'`` force GPU-accelerated execution.
+- Useful when you want to ensure the query runs on a specific engine.
 
 Integration with PyData Ecosystem
 ---------------------------------

docs/source/gfql/benchmark_filter_pagerank.rst

@@ -30,7 +30,10 @@ no database required. This benchmark compares **Graphistry's local Cypher**
      - **3.33s**
      - **>56x**
 
-*Warm median of 5 runs, 2 warmup iterations. DGX dgx-spark, GB10 GPU.*
+*Pipeline time (search + PageRank + search), warm median of 5 runs, 2 warmup iterations. DGX
+dgx-spark, GB10 GPU. The per-graph sections below report full-lifecycle totals that also include
+one-time ETL/load — hence the slightly larger numbers there (e.g. GPlus GPU 3.33s pipeline vs
+~7.1s lifecycle).*
 
 The pipeline
 ------------
@@ -173,8 +176,23 @@ pandas / cuDF). That is what makes the CPU-to-GPU switch a configuration
 flag (``engine="cudf"``) rather than a rewrite, and what keeps ETL, search,
 and analytics in the same in-process pipeline.
 
+**Same answer on every engine.** The CPU and GPU results above are not just
+comparable — they are *identical*. Differential parity across ``pandas`` /
+``polars`` / ``cudf`` / ``polars-gpu`` is a GFQL release gate: an engine either
+returns the same result or raises ``NotImplementedError`` — never a silently
+different answer. So the speedups here are a pure hardware/engine choice, not a
+change in what the query means.
+
+This page is one workload (a filter → PageRank → filter pipeline) against one
+external baseline (Neo4j+GDS). For the full four-engine picture — when Polars
+beats pandas on CPU, when the GPU pulls ahead, and how to choose — see
+:doc:`engines`. For sub-millisecond *seeded* lookups that beat Kuzu and Neo4j
+by 9–28×, see :doc:`index_adjacency`.
+
 For more on the GFQL design and supported surface:
 
+- :doc:`engines` — choosing pandas / Polars / cuDF / Polars-GPU
+- :doc:`index_adjacency` — seeded-traversal CSR adjacency index
 - :doc:`cypher` — Cypher syntax through ``g.gfql("MATCH ...")``
 - :doc:`overview` — GFQL design, features, and GPU acceleration
 - :doc:`about` — 10-minute introduction to GFQL