Repository navigation
Conversation
1eece7a to
afe8ce0
Compare
line-o
left a comment
There was a problem hiding this comment.
Resolving modules to load by default over SPI received pushback on the community call on 2026-07-06
Security concerns need to be addressed before we can consider pulling this in.
@dizzzz specific concern was: "I see autoloading modules that are not mentioned in the configuration as a risk."
My concern is: if bundled modules are left out of the configuration I have to know they are there in order to disable them again.
|
That said, I had similar ideas re module loading a long time ago. "plug the modules you only need". My original ideas can from a slightly different angle: will this make loading java modules from a XAR file more simple? |
dizzzz
left a comment
There was a problem hiding this comment.
it looks that this PR combines 2 or 3 PRs (which makes the PR more difficult to review than strictly needed).
|
These are stacked PRs that all target MutableCollection.java:106 (Namespaces.java) MutableCollection.java:116 (Caffeine / separate class) IndexManager.java:146 (null/blank id) JettyStart.java:125 (different PR) On your question about XAR module loading: |
153a184 to
11d1c92
Compare
| for (int i = 0; i < params.getLength(); i++) { | ||
| final Element param = ((Element) params.item(i)); | ||
|
|
||
| if ("no".equalsIgnoreCase(param.getAttribute("enabled"))) { |
There was a problem hiding this comment.
"no" only — @enabled's type in conf.xsd is yes_no, an enumeration restricted to exactly yes/no (not xs:boolean), so "false" isn't a valid value to begin with. All 5 call sites (4 in Configuration.java, this one) check the same literal for that reason.
9e0973c to
c03d824
Compare
📊 XQTS result comparisonComparison of this run against
Relative to Runtime: 456.6s (+62.40s vs |
c03d824 to
3227cc6
Compare
3227cc6 to
c705c26
Compare
…review Addresses findings from a full-diff code review of the ModuleFactory/IndexFactory SPI and system: configuration observability work, plus the schema-governance and Codacy issues surfaced on the PR's CI run: - Configuration.java: SPI IndexFactory auto-discovery was gated behind the pre-existing "no <modules> element" early return, so a conf.xml without an <indexer><modules> block silently got no SPI-registered indexes. The guard now only skips the conf.xml-parsing loop, not SPI discovery. - VectorEmbeddingService: replaced a non-atomic get/check/create/put with cache.computeIfAbsent() so concurrent getProviderByPath() calls for the same uncached model can't both load an ONNX model (the losing instance leaked). - ModelRegistry: configure() now writes the singleton under the same synchronized(ModelRegistry.class) lock getInstance() reads under, closing a race where a concurrent getInstance() could win and build a stale fallback registry that configure() would then silently overwrite. - GetConfigurationProperty: corrected the docs to say what the function actually does (compiles and runs $path as a full XQuery expression, not a restricted XPath subset) so the capability isn't assumed safe if ever reused for a lesser-privileged role. - IndexManager/Configuration: added an explicit source field (built-in/spi/ conf.xml) to IndexModuleConfig, mirroring ModuleRegistration, and register the always-on structural index into the registry so it shows up in system:get-registered-indexes() instead of being silently omitted. - VectorStoreServiceImpl: replaced three Class.forName().getMethod().invoke() reflection call sites with a new VectorExtensionHook SPI (ServiceLoader- discovered), mirroring the ModuleFactory/IndexFactory pattern this same PR introduced, instead of ad hoc reflection for the same "optional extension registers with core" problem. - GetRegisteredModules: dropped a throwaway XQueryContext that reflectively re-instantiated every built-in module class on each call; reuses the live calling context instead, which already has them loaded. - schema/conf.xsd + conf.xml, schema/controller-config.xsd + controller-config.xml: bumped xs:schema/@Version per schema/README.md's governance policy — conf.xsd's new <vector-models enabled> attribute is a MINOR addition, and controller-config.xml's new /schema/ forward entry required a PATCH bump on its paired schema even though the schema's own content didn't change. - SchemaServletTest: three tests only called EasyMock.verify(...), which Codacy's "JUnit tests should include assert()" check doesn't recognize; wrapped in assertThatCode(...).doesNotThrowAnyException() so the existing check is visible to the linter. ConfigurationRedactor's per-call redaction cost was investigated but left as-is: eXist's memtree NodeImpl throws UnsupportedOperationException for setUserData/getUserData, and caching a memtree Document across unrelated XQueryContext instances is unverified/risky, so no safe low-risk fix applies. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
4cf0378 to
4598cc7
Compare
|
@dizzzz go time? |
832c342 to
25464c0
Compare
…nonical conf.xml Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…abled to <vector-models> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…canonical conf.xml Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…f.xml element
AbstractIndex.configure() only sets name from config.getAttribute("id") when
config != null. IndexFactory SPI registration passes config=null, leaving name
null. IndexController.getWorkerByIndexName() matches on name, so SPI-registered
indexes (range-index, ngram-index, sort-index, lucene-index) were never found,
causing NPE at every range:index-keys-for-field() call.
Fix: after configure(), if name is still null, call setName(id) with the
configuration id that was used to key the indexers map.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…es covered by existing imports Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
factory.getDefaultId() is a third-party contract; a null or blank return would silently register the index under a useless key and leave its name unset. Validate at the Configuration.java SPI loop (primary gate) and add a belt-and-suspenders check in IndexManager.initIndex so the name is never set to null/blank. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ModuleFactory/IndexFactory SPI autodiscovery plus enabled="no" suppression opened a gap: a module or index can now be active with no conf.xml entry at all, and a suppressed entry is discarded at startup with no trace. Neither is reportable. Adds a new ModuleRegistration record and, for indexes, an `enabled` field on the existing IndexModuleConfig record, both populated alongside (not instead of) the classMap/PROPERTY_INDEXER_MODULES that actually drive module/index instantiation - so active behavior is unchanged, but every module and index now has a retained provenance record (spi/conf.xml/built-in, and whether currently suppressed) for the new system: functions added in a following commit. Configuration also now retains its parsed conf.xml Document, needed by system:get-configuration() and system:get-configuration-property(). Refs eXist-db#6563 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New DBA-only system: functions: - get-registered-modules() - util:registered-modules-info()'s content extended with registration-source (spi/conf.xml/built-in) and enabled, including enabled="no"-suppressed entries. - get-registered-indexes() - same shape for index modules; no equivalent existed before. - get-configuration() - the effective parsed conf.xml as an in-memory element, with credential-shaped attribute/element values redacted (ConfigurationRedactor), including eXist's own <parameter name="password" value="..."/> idiom. - get-configuration-schema-version($schema) - reads the SchemaVersion build-time constants by schema name. - get-configuration-property($path) - an XPath/XQuery accessor into the same redacted document get-configuration() returns, evaluated by eXist's own XQuery engine against the redacted root (the parsed conf.xml is eXist's own in-memory DOM, which standard javax.xml.xpath cannot traverse). Landed in the system: namespace per discussion thread (2026-07-08 to 2026-08-19): reusing util: was rejected as adding to an already-mixed bag, and a new config:/conf:/db: namespace was rejected as a compatibility hazard (config: in particular collides with common existing user code) and unnecessary churn - system: already hosts get-running-xqueries() and other DBA-only introspection functions. Refs eXist-db#6563 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
util:registered-modules-info(), util:mapped-modules(), and util:is-module-mapped() are now fully covered by system:get-registered-modules() (mapped-modules()/is-module-mapped() overlap it completely - same backing data, PROPERTY_STATIC_MODULE_MAP; registered-modules-info()'s content is a strict subset). Deprecated in place with no content change, so existing callers keep working unchanged; only the FunctionSignature gains a deprecation notice pointing at the replacement. Refs eXist-db#6563 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
$EXIST_HOME/schema/*.xsd is shipped in every distribution layout but was only reachable on the local filesystem. Adds SchemaServlet, wired via controller-config.xml's URL-rewriting pipeline the same way JMXServlet's /status is, serving a bare "<name>.xsd" filename under that directory over HTTP with no authentication - the schemas are plain, publicly-shippable XSDs with no sensitive content, so external tooling (IDE plugins, editors) can resolve a config file's grammar without vendoring its own copy. Guards against path traversal by rejecting any request path with more than one path segment, and independently by normalizing the resolved file path and confirming it stays under the schema directory. Refs eXist-db#6563 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…PI work Addresses findings from a full-diff code review of the ModuleFactory/IndexFactory SPI and system: configuration observability work, plus schema-governance and static-analysis issues surfaced by CI: - Configuration.java: SPI IndexFactory auto-discovery was gated behind the pre-existing "no <modules> element" early return, so a conf.xml without an <indexer><modules> block silently got no SPI-registered indexes. The guard now only skips the conf.xml-parsing loop, not SPI discovery. - VectorEmbeddingService: replaced a non-atomic get/check/create/put with cache.computeIfAbsent() so concurrent getProviderByPath() calls for the same uncached model can't both load an ONNX model (the losing instance leaked). - ModelRegistry: configure() now writes the singleton under the same synchronized(ModelRegistry.class) lock getInstance() reads under, closing a race where a concurrent getInstance() could win and build a stale fallback registry that configure() would then silently overwrite. - GetConfigurationProperty: corrected the docs to say what the function actually does (compiles and runs $path as a full XQuery expression, not a restricted XPath subset) so the capability isn't assumed safe if ever reused for a lesser-privileged role. - IndexManager/Configuration: added an explicit source field (built-in/spi/ conf.xml) to IndexModuleConfig, mirroring ModuleRegistration, and register the always-on structural index into the registry so it shows up in system:get-registered-indexes() instead of being silently omitted. - registered-indexes.xqm: updated invalid-registration-source's allowed values to include "built-in" and added a dedicated assertion that structural-index reports registration-source "built-in" and enabled "yes" - the old test predated the structural-index registry fix above and failed once it always appeared. - VectorStoreServiceImpl: replaced three Class.forName().getMethod().invoke() reflection call sites with a new VectorExtensionHook SPI (ServiceLoader- discovered), mirroring the ModuleFactory/IndexFactory pattern, instead of ad hoc reflection for the same "optional extension registers with core" problem. - GetRegisteredModules: dropped a throwaway XQueryContext that reflectively re-instantiated every built-in module class on each call; reuses the live calling context instead, which already has them loaded. - schema/conf.xsd + conf.xml, schema/controller-config.xsd + controller-config.xml: bumped `xs:schema/@version` per schema/README.md's governance policy: conf.xsd's new <vector-models enabled> attribute is a MINOR addition, and controller-config.xml's new /schema/ forward entry required a PATCH bump on its paired schema even though the schema's own content didn't change. - SchemaServletTest: three tests only called EasyMock.verify(...), which Codacy's "JUnit tests should include assert()" check doesn't recognize; wrapped in assertThatCode(...).doesNotThrowAnyException() so the existing check is visible to the linter. - VectorExtensionHook: documented the three intentionally-empty default method bodies (Codacy: "Document empty method body"). - VectorExtensionLifecycle: dropped the explicit no-arg constructor in favor of the implicit one ServiceLoader already uses (Codacy: "Avoid unnecessary constructors"). ConfigurationRedactor's per-call redaction cost was investigated but left as-is: eXist's memtree NodeImpl throws UnsupportedOperationException for setUserData/getUserData, and caching a memtree Document across unrelated XQueryContext instances is unverified/risky, so no safe low-risk fix applies. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
VectorStoreServiceImpl discovers VectorExtensionLifecycle via ServiceLoader (META-INF/services/org.exist.storage.vector.VectorExtensionHook) instead of the Class.forName().getMethod().invoke() reflection it previously used. That dispatch path had no dedicated regression test: VectorOperationMetricsTest and VectorEmbeddingJmxTest exercise the hooks' own behavior, but only incidentally prove the SPI discovery works, and only for the startup half of the lifecycle. Verified the new test actually catches the regression it targets: temporarily removed the META-INF/services registration (source file and stale target/classes copy) and confirmed both tests fail with the bridge never getting registered, then restored the file and confirmed the suite is green again. Add VectorExtensionHookWiringTest with two tests that only pass if the SPI dispatch fires automatically through a real BrokerPool boot/shutdown - neither test ever calls VectorExtensionLifecycle directly: - startupHookRegistersMetricsBridgeWithoutManualWiring: recordEmbed() reaches the metrics bridge immediately after boot. - shutdownHookUnregistersMetricsBridgeOnRealBrokerPoolShutdown: after a real stopDb(), recordEmbed() becomes a silent no-op, proving the shutdown hook actually unregistered the bridge rather than it merely still being absent. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…nsionHook tradeoff Addresses two follow-ups from a senior-level review of the module/index SPI work: - GetRegisteredModules and ModuleInfo (util:registered-modules(), util:registered-modules-info()) independently walked the same three module sources (built-in, EXPath package, conf.xml-mapped) with their own copies of the dedup/prefix-lookup logic. ModuleInfo's copies also still had the throwaway-XQueryContext performance bug already fixed in GetRegisteredModules (reflectively re-instantiating every built-in module class per call), since fixing it there didn't touch ModuleInfo's separate implementation. - Extracted the walk into org.exist.xquery.LiveModules.collect(XQueryContext), colocated with Module/XQueryContext/ModuleRegistration rather than under either function package, since util:'s deprecated functions depending on system:'s implementation classes would be an odd direction. Both callers now map the same List<LiveModules.Entry> into their own output shape; the perf fix now benefits both instead of only the one that was touched directly. Net effect: -128 lines of duplicated logic, all three functions' behavior verified unchanged (UtilTests 69/69, SystemTests 36/36). - VectorExtensionHook: added a design-note doc comment explaining why this is a separate, narrower SPI rather than making BrokerPoolService itself ServiceLoader-discoverable (which already has the same configure/startSystem/ shutdown shape) - broader change to core startup sequencing than warranted for replacing one extension's reflection. Left as a documented tradeoff, not built speculatively; flagged as the thing to revisit if a second optional extension needs the same trick. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018PcvseE618pZu4eVbKtCW5
25464c0 to
da2859f
Compare
Summary
ServiceLoader-based SPI for XQuery modules and index modules, so bundled implementations are auto-discovered without explicitconf.xmlentries. New installs get a leaner default config; existing hand-maintained configs are unaffected — explicit entries always win over SPI, andenabled="no"(from #6550) suppresses an SPI entry without removing the JAR.Also delivers the XQuery observability API this SPI/
@enabledwork created a need for (#6563): with modules and indexes now able to be active with noconf.xmlentry at all, or silently suppressed, the DBA needs a way to ask "what's loaded and why."Closes #3062 — delivers both parts: the
@enabledattribute (#6550) and the SPI/CDI-style autodiscovery this PR adds.Closes #6563 — the observability API this and #6550 created a need for.
What changed
ModuleFactorySPI — neworg.exist.xquery.ModuleFactoryinterface;Configuration.configureModules()scansServiceLoader<ModuleFactory>before theconf.xmlloop. All 27 bundled XQuery modules register viaMETA-INF/services/.IndexFactorySPI — neworg.exist.indexing.IndexFactoryinterface;IndexManagerregisters SPI entries whoseidhas no liveconf.xmlentry. All bundled indexes (Lucene, ngram, range, sort, spatial) wired; indexnamedefaults to the SPIidwhen there's noconf.xmlelement to supply one; guarded against a null/blankidfrom the factory.Vector model registry — integrated with
Configuration;@enabledon<vector-models>children; repeated WARN for unavailable models downgraded to DEBUG after the first occurrence.exist-distribution/src/main/config/conf.xml—<builtin-modules>and index<modules>sections trimmed; the 27 bundled modules and 5 bundled indexes no longer listed individually.Module/index registration provenance — new
ModuleRegistrationrecord and anenabledfield on the existingIndexModuleConfigrecord, populated alongside (not instead of) the active classMap/PROPERTY_INDEXER_MODULES— so instantiation behavior is unchanged, but every module and index now carries a retained provenance record (spi/conf.xml/built-in, and whether currently suppressed).New
system:observability functions (DBA-only) — landed in thesystem:namespace per discussion on #6563 (2026-07-08 → 2026-08-19): reusingutil:was rejected as adding to an already-mixed bag, and a newconfig:/conf:/db:namespace was rejected as a compatibility hazard and unnecessary churn.system:get-registered-modules()—util:registered-modules-info()'s content extended withregistration-source(spi/conf.xml/built-in) andenabled, includingenabled="no"-suppressed entries.system:get-registered-indexes()— same shape for index modules; no equivalent existed before.system:get-configuration()— the effective parsedconf.xmlas an in-memory element, with credential-shaped attribute/element values redacted (including eXist's own<parameter name="password" value="..."/>idiom).system:get-configuration-schema-version($schema)— reads theSchemaVersionbuild-time constants by schema name.system:get-configuration-property($path)— an XPath/XQuery accessor into the same redacted document, evaluated by eXist's own XQuery engine.Deprecated
util:functions —util:registered-modules-info(),util:mapped-modules(),util:is-module-mapped()deprecated in place (no content change) in favor ofsystem:get-registered-modules().Schema HTTP endpoint — new
SchemaServletserves$EXIST_HOME/schema/*.xsdunauthenticated at/exist/schema/{name}.xsd(wired viacontroller-config.xml, same pattern as/status→JMXServlet), so external tooling (IDE plugins, editors) can resolve a config file's grammar without vendoring its own copy.Compatibility
Explicit
conf.xmlentries always take precedence over SPI discovery for the same namespace URI/indexid— no upgrade action required.enabled="no"suppresses a bundled module/index without removing its JAR.The three deprecated
util:functions keep their exact current behavior — deprecation is metadata-only, not a breaking change.Test plan
mvn validate(repo-wide) — schema governance / canonical instance validation passesmvn test -pl exist-core -Dtest="xquery.system.SystemTests"— newsystem:observability function XQSuite coverage passesmvn test -pl exist-core -Dtest="xquery.util.UtilTests"— existingutil:coverage passes unchanged (deprecation is non-breaking)mvn test -pl exist-core -Dtest="org.exist.util.ConfigurationTest,org.exist.xquery.functions.system.ConfigurationRedactorTest,org.exist.http.servlets.SchemaServletTest"— registry provenance (incl. an injectedenabled="no"module/index), credential redaction, andSchemaServletpath-traversal/404 handling.codacy/cli.sh) clean on all new/touched filesenabled="no"on a module's conf.xml entry → not loaded despite SPI discovery, but reported (enabled: "no") bysystem:get-registered-modules()curl http://localhost:8080/exist/schema/conf.xsdreturns the XSD unauthenticated🤖 Generated with Claude Code