perf(gfql/index): index-accelerated get_degrees + EXISTS-prune (#5 degree-cache / #3 membership)#1697
Open
lmeyerov wants to merge 7 commits into
Open
perf(gfql/index): index-accelerated get_degrees + EXISTS-prune (#5 degree-cache / #3 membership)#1697lmeyerov wants to merge 7 commits into
lmeyerov wants to merge 7 commits into
Conversation
lmeyerov
force-pushed
the
feat/gfql-graphframes-bench
branch
from
05bdb24 to
3d0a0c1
Compare
…membership) Read node degrees straight from a resident CSR adjacency index (degree = diff(group_offsets), gathered per node via searchsorted) instead of the O(E) group_by(endpoint)+join. Bulk over all nodes -> always profitable when a valid index is resident; index_policy='off' skips; try/except falls back to the scan path (never wrong). New graphistry/compute/gfql/index/degrees.py (degrees_from_index), engine-native (numpy pandas/polars, cupy cudf). Wired into ComputeMixin.get_degrees (pandas/cudf). Prototype (1M/5M pandas): exists_prune 342->2.7ms (128x), q8 541->0.8ms (664x), exact parity. Unit + end-to-end parity green (arbitrary node ids + self-loops). WIP: polars get_degrees_polars wiring + index-suite parity tests + 4-engine dgx. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Same resident-CSR fast path (degrees_from_index) in get_degrees_polars — eager frames only (LazyFrame would force a collect), policy 'off' skips, try/except falls back to the group_by scan. Parity green vs group_by baseline (degree / degree_in / degree_out, arbitrary node ids + self-loops). Covers polars + polars-gpu (numpy-host arrays). cudf/polars-gpu GPU parity pending dgx verify. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Parity vs group_by scan + self-loop/isolated contract + index_policy='off' fallback, engine-parametrized. Local: pandas/polars/polars-gpu pass; cudf needs GPU (cupy init) — verified on dgx. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…membership) Bare EXISTS pattern (no endpoint/edge filters, no drop-self neq) = "node has an edge in this direction" = CSR adjacency membership. Route _pattern_alias_keys_polars through adjacency_membership_keys (union of edge_out/in_adj keys_sorted for undirected; per-direction otherwise) instead of the O(E) chain_polars; strict guard falls through for any richer shape. New adjacency_membership_keys in index/degrees.py. Parity + policy-off tests. Perf (200k/800k, polars): scan 1206ms -> index 16.5ms (73x); parity exact incl self-loop-only nodes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lmeyerov
force-pushed
the
feat/gfql-index-degrees
branch
from
cd6f8ae to
3b6b855
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1668 (→ #1661 → #1658 → master). First L4 index-acceleration unit off the #1658 CSR registry.
What
Node degrees + has-edge membership fall straight out of the resident CSR adjacency index (
degree = diff(group_offsets), membership =keys_sorted), turning O(E)group_by+join into an O(N)searchsortedgather.index/degrees.py—degrees_from_index(per-node in/out degree, engine-native searchsorted, arbitrary node ids) +adjacency_membership_keys(Document API with pydoc #3, per-direction / undirected union).get_degreesfast path —ComputeMixin.get_degrees(pandas/cudf) +get_degrees_polars(polars/polars-gpu). Feeds q8-style degree-products and any degree analytics / CALL.EXISTS { (n)--() }prune-isolated —_pattern_alias_keys_polarsroutes the bare pattern through membership instead ofchain_polars.index_policy='off'skips;try/except→ scan fallback (never wrong); strict guards decline any filtered/neq/multi-hop shape.Perf (parity-exact throughout)
EXISTS {(n)--()}scan 1206ms → index 16.5ms (73×); scales to sub-second at the 10m receipt (was 35s).Tests
graphistry/tests/compute/gfql/index/test_index.py: get_degrees parity ×4 engines + self-loops/isolated +policy='off'; exists_prune membership parity + policy-off. Local non-GPU green (56 index tests); cudf/polars-gpu verified on dgx.Notes / follow-ups
semi_apply_mark) still on the scan path — follow-up.bindingsto refer to them instead #2 per-column value index (panel_filters), Replace NaNs with nulls since node cannot parse JSON with NaNs #1 trigram (search_edges).expr-existentialsubquery1-1from feat(gfql/cypher): EXISTS { pattern } subqueries — declarative prune-isolated, native on 4 engines #1679; polarsfloorconformance).🤖 Generated with Claude Code