✨ Add execute-query MCP Tool (overture-stack/admin#177)#1077
Conversation
- Added `modules/sqon` (`@overture-stack/sqon`) as a dependency to `mcp-server` - Updated `mcp-server` Dockerfile build target to copy `modules/sqon` dependencies - Renamed `ArrangerIntrospectionClient` to `ArrangerClient` to reflect full capabilities (more than just introspection, now allows executing queries) - Added `executeQuery` to `ArrangerClient`, enabling making `POST` requests containing GQL queries to Arranger - Added `arranger/queryBuilder.ts` to MCP Server, containing functions for building parameterized GraphQL requests for Arranger catalogue queries, and accompanying unit tests - Added `arranger/queryValidation.ts` to MCP Server, containing functions for validating the different parts of Arranger GraphQL queries (`sqon`, `hits` fields, `aggregations` fields, `sort` fields), and accompanying unit tests - Added `arranger/queryResults.ts` to MCP Server, which takes an Arranger GQL hits response and turns it into flat, plain documents that are more easily read by an LLM. - Registered new `execute-query` tool to MCP Server, which builds, validates, confirms, and executes a SQON-filtered GQL query against one Arranger catalogue - Updated existing MCP Server integration tests to account for new `execute-query` tool - Added new integration tests for MCP Server `execute-query` tool
| **Fix:** (a) Docs/schema comments pass: update README.md:13, docs/usage/02-arranger-components.md (title + line 29), configs.json.schema:28, and console strings in configs/index.ts. (b) Identifier rename pass (separate commit): buildCatalogsFromFolder -> buildCatalogsFromDirectory, folderName -> directoryName in apps/search-server/src/configs/. (c) Cross-references: add pointer to docs/concepts.md early in docs/usage/02-arranger-components.md; introduce "filter clause" for leaf nodes in docs/sqon/03-sqon-in-detail.md. | ||
| **Standalone:** yes — (a) is docs-only; (b) is a mechanical rename; (c) is a docs addition. All three independent. | ||
|
|
||
| ### `docs/concepts.md` SQON examples use `field` instead of `fieldName` |
There was a problem hiding this comment.
Small part of why LLMs might be failing to produce valid SQON at the moment. If these docs were in any of their training data, there's a chance the LLM falls back to this and that's why I'm seeing field being used instead of fieldName.
I plan on opening a supplemental PR updating this, plus a few other enhancements like including "few-shot" examples of valid SQON in the prompt and tool descriptions, to see if that improves SQON generation on small local LLMs.
| **Fix:** Consolidate into `modules/sqon` as the single source of truth. Extend `getSqonFieldOperatorDetails()` to carry the same field-type classification detail that `buildCatalogueIntrospection.ts` currently encodes locally. `buildCatalogueIntrospection.ts` then becomes a thin projection over the module's data. See [roadmap: consolidate field-type-to-operator rules](roadmap.md#consolidate-field-type-to-operator-rules-into-modulessqon). | ||
| **Standalone:** yes — internal refactor, no change to API output | ||
|
|
||
| ### Published SQON JSON Schema contains dangling `$ref` pointers after `anyOf` → `oneOf` normalization |
There was a problem hiding this comment.
This was something Claude identified as a potential bug/reason for why LLMs might not be generating the correct SQON structure. Basically, small local LLMs aren't nesting leaf nodes correctly, resulting in something like this:
// invalid SQON
{
"op": "and",
"content": [
{
// "field" instead of "fieldName", see previous comment
"field": "donors.gender",
"op": "in",
// missing leaf node of SQON
"value": [
"Female"
]
}
]
}
Instead of something like this:
// valid sqon
{
"op": "and",
"content": [
{
"op": "in",
"content": {
"fieldName": "donors.gender",
"value": [
"Female"
]
}
}
]
}
|
|
||
| ## graphql-router | ||
|
|
||
| ### `initializeSets` startup race breaks one catalogue's GraphQL endpoint in multicatalog mode |
There was a problem hiding this comment.
This was flagged by Claude while writing integration tests. I haven't been able to reproduce it myself, but I'm also not very familiar with the graphql-router module. Not sure how much stock we want to put into this, happy to downgrade the severity or remove this if needed.
There was a problem hiding this comment.
funny enough, I just ran across this one, so I'm surfacing it as a higher priority on my end! thanks for flagging it here 🙏
There was a problem hiding this comment.
Ha i ran into this too! please fix sir Justin, it happens more when I am loading an arranger server with 3-5 catalogues
| errors?: { message: string; [key: string]: unknown }[]; | ||
| }; | ||
|
|
||
| export type ArrangerClient = { |
There was a problem hiding this comment.
Renamed ArrangerIntrospectionClient to ArrangerClient since it now handles more than just Arranger introspection (i.e. also executes GraphQL queries now)
| * }); | ||
| * ``` | ||
| */ | ||
| export const buildArrangerGraphQLQuery = (input: BuildArrangerGraphQLQueryInput): ArrangerGraphQLRequest => { |
There was a problem hiding this comment.
Main function in queryBuilder.ts. The purpose of this function (and this file) is to handle the deterministic creation of a GQL query we can send to Arranger, given inputs to the execute-query tool from an LLM.
The execute-query tool validates the inputs itself (using functions from queryValidation.ts) prior to this function being called.
| /** | ||
| * Strips the GraphQL `edges`/`node` nesting from an Arranger hits response, including the | ||
| * connection wrappers that `nested` fields carry inside each hit node. | ||
| * @param edges - The `hits.edges` array from an Arranger GraphQL response. | ||
| * @param fieldTypes - Map of dot-notation field name to its introspected field type. | ||
| * @returns One flat document object per hit. | ||
| * @remarks This makes the response easier to work with for LLMs. | ||
| * @example | ||
| * ```ts | ||
| * compactHitNodes({ | ||
| * edges: [{ node: { id: 'f1', donors: { hits: { edges: [{ node: { age: 41 } }] } } } }], | ||
| * fieldTypes: { donors: 'nested', 'donors.age': 'long' }, | ||
| * }) | ||
| * // returns [{ id: 'f1', donors: [{ age: 41 }] }] | ||
| * ``` | ||
| */ | ||
| export const compactHitNodes = ({ |
There was a problem hiding this comment.
Intended to flatten/shrink the size of responses to make them more manageable for LLMs and reduce the amount of tokens needed to process the result of Arranger queries.
This wasn't a requirement in the ticket, but came up when I was trying to optimize things last week.
There was a problem hiding this comment.
This queryValidation.ts file contains a bunch of small validation functions used by the execute-query tool.
| const inputSchema = { | ||
| catalogueId: zod.string().min(1).describe('Catalogue identifier from the Arranger /introspection payload.'), | ||
| sqon: zod | ||
| .unknown() |
There was a problem hiding this comment.
Area for improvement in that follow-up PR I mentioned in my first comment: provide a Zod schema for SQON here (rather than unknown) so the tool description contains that. Would greatly improve the chances that an LLM generates valid SQON.
Could be an area for potential drift in the future, though. If a schema provided here doesn't align with what is returned from the get-sqon-schema tool.
| sqon: zod | ||
| .unknown() | ||
| .describe( | ||
| 'SQON filter for the query. Required. For an unfiltered query ("show me everything"), pass the empty root SQON {"op": "and", "content": []} — never null. Use get-sqon-schema for the SQON structure and get-catalogue-fields for valid fields and per-type operators.', |
There was a problem hiding this comment.
If providing a Zod schema for SQON to the tool input doesn't seem like a good idea due to drift, another enhancement would be to include "few-shot" examples of valid SQON in the description here.
| COPY --from=scaffolding --chown=$APP_USER:$APP_USER \ | ||
| $APP_FOLDER/modules/sqon/dist \ | ||
| ./modules/sqon/dist | ||
| COPY --from=scaffolding --chown=$APP_USER:$APP_USER \ | ||
| $APP_FOLDER/modules/sqon/package.json \ | ||
| ./modules/sqon/package.json |
There was a problem hiding this comment.
Copying the SQON module into the mcp-server image, now that it is a dependency. Similar to how it is handled for search-server
* Fix spelling of `catalog` --> `catalogue` inside "List Arranger Catalogues" tool result * Add missing newlines to end of unit test files, minor prettier formatting changes
* `ArrangerRequestError` carries `status`, `statusText`, `body`, and an `isTimeout` flag so that appropriate, actionable errors can be returned * `execute-query` tool returns formatted, actionable errors, including previously uncaught Zod parse calls and transport failures
justincorrigible
force-pushed
the
main
branch
4 times, most recently
from
082f8f1 to
5440566
Compare
3 participants
Summary
Adds
execute-querytool to MCP Server, which enables making SQON-filtered GraphQL requests to Arranger.Issues
Description of Changes
MCP Server
modules/sqon(@overture-stack/sqon) as a dependency tomcp-servermcp-serverDockerfile build target to copymodules/sqondependenciesArrangerIntrospectionClienttoArrangerClientto reflect full capabilities (more than just introspection, now allows executing queries)executeQuerytoArrangerClient, enabling makingPOSTrequests containing GQL queries to Arrangerarranger/queryBuilder.tsto MCP Server, containing functions for building parameterized GraphQL requests for Arranger catalogue queries, and accompanying unit testsarranger/queryValidation.tsto MCP Server, containing functions for validating the different parts of Arranger GraphQL queries (sqon,hitsfields,aggregationsfields,sortfields), and accompanying unit testsarranger/queryResults.tsto MCP Server, which takes an Arranger GQL hits response and turns it into flat, plain documents that are more easily read by an LLM.execute-querytool to MCP Server, which builds, validates, confirms, and executes a SQON-filtered GQL query against one Arranger catalogueIntegration Tests
execute-querytoolexecute-querytoolSpecial Instructions
Before running these changes, you will need to:
# from project root npm run modules:buildmodules/sqontomcp-server):# from project root npm ciReadiness Checklist
.env.schemafile and documented in the README