Add NetworkAggregation mode to Charts by joneubank · Pull Request #1076 · overture-stack/arranger

joneubank · 2026-06-15T20:57:59Z

Summary

Adds a mechanism to have the Chart components perform a network aggregation search and display that network aggregation data.

Each of the existing chart types (BarChart and SunburstChart) have a new property isNetworkAggregation that if set to true will switch their functionality to use the network aggregation search query.

A new chart type <NetworkNodesChart /> is also added to query specifically the hits on each node in the Arranger search network. This chart is a BarChart with data handling dedicated to the total hits on each node in the network.

Lastly, a mechanism was added to the graphql-router to filter the network query to only include a subset of the available nodes. This mechanism is a nodesFilter value that is provided to the network type in the GQL query. It filters based on the included nodeIds. To support this change, the configuration of the arranger server now requires each node in the network to have a nodeId and the arranger server application was updated to read this value in its configs.

Readiness Checklist

Self Review
- I have performed a self review of code
- I have run the application locally and manually tested the feature
- I have checked all updates to correct typos and misspellings
Formatting
- Code follows the project style guide
- Autmated code formatters (ie. Prettier) have been run
Local Testing
- Successfully built all packages locally
- Successfully ran all test suites, all unit and integration tests pass
Updated Tests
- Unit and integration tests have been added that describe the bug that was fixed or the features that were added
Documentation
- All new environment variables added to .env.schema file and documented in the README
- All changes to server HTTP endpoints have open-api documentation
- All new functions exported from their module have TSDoc comment documentation

Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com>

* Use SearchQueryResponse * createElasticClient * Types file updated * createOpenSearch updates * Use overture-stack/types import * SharedIndicesCreateParams type change * Remove yarn.lock * Update parameter types * Update Response Types * Minor edits * Search Params & Body updates, removed instance of Prettify, Update Import names * Revise Options & Config types * Minimal Search Options type * Consistent Naming

* Rename network config as nodes, remove config property to enable network search * search-server remove env flag for network search * Types clean up and improved nodes data reporting in network query - nodes hits now return if no aggregations are requested - nodes that failed to connect on startup are included in the nodes data query - overhauled the Result type to allow different data to be returned for each result case - created a seaprate config type for local vs remote nodes, not yet used * WIP: Enhanced types for graphql-router and network search on local node - Next Step is to define local node configs and build the local query from those - Converted all graphql-router/schema files to typescript - Added Context as a generic type for many of the Resolver types and related functions. This is used to pass information to the resolvers and custom resovler functions. At the moment this is only the server side filter function, but passing custom context to this is important. * Working network search within a catalog * Move shared types into dedicated files * Update config schema to match new config format * validateConfigs will list all missing properties from the config

@justincorrigible

suggestion from @justincorrigible Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com>

@justincorrigible

suggestion from @justincorrigible Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com>

…components

joneubank · 2026-06-15T21:45:28Z

Branch was originally made off of a previous PR and so it has a long list of old commits that I haven't cleared out. New changes begin with 58f714a .

joneubank · 2026-06-15T22:00:24Z

+		getNetworkChartData,
+		getNetworkNodesData,
+		requireNetworkSearch,


Use these new functions exported from the provider to access network search data, or require the network search to run.

If a chart uses network search, this can be specified as an additional property in the existing registerChart function.

joneubank · 2026-06-15T22:01:14Z


 export interface BarChartProps {
 	fieldName: string;
+	isNetworkAggregation?: boolean;


This new property is the key addition, this will be set to true to indicate that the BarChart should the data from the network aggregation search instead of the default local catalog aggregation data.

justincorrigible · 2026-06-16T21:41:44Z

+		: useMemo(
+				() => validateQueryProps({ fieldName, variables, extendedMapping }),
+				[fieldName, variables, extendedMapping],
+			);


MUST for this PR.

think this will error out as a Rules of Hooks validation. Hooks should be called unconditionally.

odd, I thought we had React ESLint to help flag these out in this repo. TODO for myself to make sure they're working.

justincorrigible · 2026-06-16T21:42:15Z

+		: useMemo(
+				() => validateQueryProps({ fieldName, variables: {}, extendedMapping }),
+				[fieldName, extendedMapping],
+			);


MUST for this PR.

same note as in barchart about Rules of Hooks conditional call error.

justincorrigible · 2026-06-16T21:46:21Z

+				// Match by nodeId.
+				// If there are nodes with duplicate nodeId then we can't differentiate, we will use the first match from the configs
+				const matchingConfig = remoteNodeExtendedConfigs.find((node) => {
+					return node.nodeId === params.remoteNode.nodeId;


MUST for this PR.

if nodeId is ever absent on both sides, then undefined === undefined => true, so anyone with multiple remote nodes and custom per-node headers but no nodeId in their config will have the first node's headers applied to all nodes.

I think this is a regression from your work in #1066, and a quick fix would be to make nodeId required in RemoteNodeConfig (which makes it easier to detect on boot), or to keep the old graphqlUrl + displayName match as fallback when nodeId is absent.

your call 👍

justincorrigible · 2026-06-16T21:50:07Z

 		"noUnusedParameters": true,
 		"noFallthroughCasesInSwitch": true,
 		"noUncheckedSideEffectImports": true,
+		"noImplicitAny": false,


COULD for this PR. Let's talk

wouldn't it be better if we forced adding any rather than allowing implicit ones that make things harder to refactor or debug later?
I have a feeling this is for the nivo configs, which could be handled with e.g. Partial<BarSvgProps> or @nivo/bar

justincorrigible · 2026-06-16T21:51:44Z

+	theme,
+}: NetworkNodesChartProps) => {
+	// ensure maxBars is provided
+	if (!maxBars) {


COULD for this PR. Let's talk

this defaults to infinity, so it will (almost?) never trigger. this condition/error handler can be removed and teh default documented in JSDoc instead

justincorrigible · 2026-06-16T21:54:35Z

+				// Need to rebuild the Map to ensure we get the state update
+				const newValuesMap = new Map(prev);
+				newValuesMap.delete(fieldName);
+				logger.debug(`Registered Chart for Network Aggregation on field '${fieldName}'`);


COULD for this PR. maybe I'm misreading it

looks like this log comes up in the removeQuery so maybe it should say Deregistered instead?

justincorrigible · 2026-06-16T21:56:58Z

 	"compilerOptions": {
 		"tsBuildInfoFile": "./node_modules/.tmp/tsconfig.app.tsbuildinfo",
-		"target": "ES2020",
+		"target": "ES2023",


TODO for another PR.

I think this was to enable Array.toSorted?
it's raising the minimum required runtime, which I'm fine with but could use being documented both in changelog and docs.

Do we currently have this documented in the docs? Just looked it up and this would put the minimum Node version to 18... but this runs in the browser so its compatible with all modern browsers either way and will likely go through a bundling process.

Yes this was for .toSorted to remove the TS error being called out in the Bar chart where it was already used, and I reused it in the NetworkNodes chart as well.

justincorrigible · 2026-06-16T22:00:30Z


 export type BaseNodeConfig = {
 	[baseNodeProperties.DISPLAY_NAME]: string;
+	[baseNodeProperties.NODE_ID]?: string;


TODO for another PR. perhaps

While making this optional is honest to implementation, but basically required for NetworkNodesChart and the filtering to work, it may make sense to at least give a warning on boot that a missing nodeId may break things or something like that

justincorrigible · 2026-06-16T22:23:50Z

+	isNetworkAggregation = false,
+	networkAggregationType = aggregationsTypenames.Aggregations,


NO Actionable. just design debate

these two could be unified into a single one, where since networkAggregationType is already optional, whenever present it's as if the isNetworkAggregation == true

True for the moment. I have ambition to have network search properly support numeric aggregations as well.

justincorrigible and others added 30 commits March 5, 2026 15:04

add SQON module, define schema and types

753de6c

abstract server and types from main Arranger module

408bd6d

add Arranger Charts

b60de0a

Co-authored-by: Ciaran Schutte <ciaranschutte@oicr.on.ca> Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com>

restore integration tests functionality

0035c8b

add better CORS management

3d53b80

add introspection endpoints

901fc9a

add turborepo for multiversion pipeline support

aa815c2

fix build integration issues

4919f73

Merge branch 'multiversion' into es-multiversion

ff24372

Merge branch 'searchengines' into 3.1

695a404

add MCP server foundation

c74cc0c

add multicatalog functionality (#1049)

c5d5551

improve multicatalog scaffolding, add testing for it

f03738a

Merge branch 'release-charts' into 3.1

ffc1f14

remove baseUrl from tsconfigs for TS@7

7415c9c

#1055: add "select all" button to facet panel (#1056)

c2435d3

Optional customizable headers in remote node network requests

9c4f52a

Optional customizable headers in remote node network requests

56a176e

Merge branch 'main' into feat/custom-remote-node-requests

42c1bab

Remove unintended logging text changes

da84267

Restore unintentional formatting change

6f6f423

Fixing merge errors

b48c448

add TODO comment

d43d605

passthroughHeaders template as empty array

3206dd0

suggestion from @justincorrigible Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com>

Remove unnecessary undefined check

42b67ad

suggestion from @justincorrigible Co-authored-by: Anders Richardsson <2107110+justincorrigible@users.noreply.github.com>

Fix merge conflicts

587535f

Allow network config to customize requests per remote node

ac23159

Merge branch 'main' into feat/custom-remote-node-requests

c3db80b

joneubank added 5 commits June 15, 2026 16:27

Add networkAggregation mode to arranger charts

58f714a

NetworkNodesChart and isNetworkAggregation option for existing Chart …

483ced7

…components

Add nodeId to network node configs

8cf35a9

Add nodesFilter arg and nodeId to network query

ba4583b

Merge branch 'main' into feat/charts-with-network-aggregation

908e154

joneubank marked this pull request as ready for review June 15, 2026 21:56

joneubank requested a review from justincorrigible June 15, 2026 21:56

joneubank commented Jun 15, 2026

View reviewed changes

Merge branch 'main' into feat/charts-with-network-aggregation

db04568

justincorrigible reviewed Jun 16, 2026

View reviewed changes

Comment thread modules/graphql-router/src/network/typeDefs/aggregations.ts

justincorrigible reviewed Jun 16, 2026

View reviewed changes

Comment thread modules/charts/src/components/charts/Bar/BarChart.tsx

justincorrigible force-pushed the main branch 4 times, most recently from 082f8f1 to 5440566 Compare June 19, 2026 03:51

Merge branch 'main' into feat/charts-with-network-aggregation

7861460

		isNetworkAggregation = false,
		networkAggregationType = aggregationsTypenames.Aggregations,

Uh oh!

Conversation

joneubank commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Readiness Checklist

Uh oh!

joneubank commented Jun 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justincorrigible Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

MUST for this PR.

Uh oh!

justincorrigible Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

MUST for this PR.

Uh oh!

justincorrigible Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

MUST for this PR.

Uh oh!

Choose a reason for hiding this comment

COULD for this PR. Let's talk

Uh oh!

justincorrigible Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

COULD for this PR. Let's talk

Uh oh!

Choose a reason for hiding this comment

COULD for this PR. maybe I'm misreading it

Uh oh!

Choose a reason for hiding this comment

TODO for another PR.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

TODO for another PR. perhaps

Uh oh!

Uh oh!

Choose a reason for hiding this comment

NO Actionable. just design debate

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

joneubank commented Jun 15, 2026 •

edited

Loading

justincorrigible Jun 16, 2026 •

edited

Loading

justincorrigible Jun 16, 2026 •

edited

Loading

justincorrigible Jun 16, 2026 •

edited

Loading

justincorrigible Jun 16, 2026 •

edited

Loading