Skip to content

Adds experimental auto-instrumentation for Spring AI 2.x. - #4622

Draft
hegerchr wants to merge 15 commits into
elastic:mainfrom
hegerchr:spring-ai-instrumentation
Draft

hegerchr wants to merge 15 commits into
elastic:mainfrom
hegerchr:spring-ai-instrumentation

Conversation

@hegerchr

@hegerchr hegerchr commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds experimental auto-instrumentation for Spring AI versions 2.0.0 through 2.0.2.

The new plugin creates spans for:

  • Synchronous and streaming chat model calls
  • Embedding model calls
  • Vector store add, delete, and similarity-search operations
  • Tool callback execution

Chat spans capture GenAI attributes including the operation, request and response models, request parameters, response metadata, finish reasons, streaming state, and token usage. Provider information is recorded only when it can be obtained reliably; it is not inferred from implementation class names.

Streaming spans remain open until the response publisher terminates, so their duration represents the overall response time. Response chunks are aggregated so completed assistant content, including tool calls and tool responses, can be captured correctly.

For streaming chat calls, time to first chunk is recorded as:

  • Span attribute: gen_ai.response.time_to_first_chunk
  • OpenTelemetry histogram: gen_ai.client.inference.time_to_first_chunk, in seconds

Content capture is disabled by default. The new dynamic capture_spring_ai_content configuration option enables capture of:

  • Prompt and response messages
  • Assistant tool calls and tool responses
  • Embedding inputs
  • Vector store documents, queries, and filters
  • Tool arguments and results

The option is documented as introduced in Java agent 1.58.0. Captured values are subject to long_field_max_length. The documentation also describes the security, performance, network, and storage implications of capturing AI content.

The setting can be configured using:

elastic.apm.capture_spring_ai_content=true

or:

ELASTIC_APM_CAPTURE_SPRING_AI_CONTENT=true

The agent supports changing this option dynamically. Configuring it through Kibana central configuration requires corresponding Kibana changes and will be supported in a future Kibana version.

The instrumentation is experimental and disabled by default. It can be enabled using:

elastic.apm.enable_experimental_instrumentations=true

The shared classloader matcher requires a marker class introduced in Spring AI 2.0, preventing unsupported Spring AI 1.x versions from being instrumented. Isolated version-compatibility tests verify that:

  • Spring AI 1.0.8 and 1.1.7 are rejected
  • Spring AI 2.0.0 and 2.0.2 are accepted

The compatibility tests use the repository’s TestClassWithDependencyRunner pattern, keeping version-specific dependencies isolated from the plugin’s normal test classpath.

Testing performed:

  • 11 Spring AI instrumentation unit tests passed
  • Four isolated version-compatibility cases passed
  • ConfigurationExporterTest passed and regenerated the configuration reference
  • Full 209-module agent build completed successfully with Java 17
  • Packaged agent startup smoke-tested with Java 17
  • End-to-end testing performed with start-local

Checklist

  • This is an enhancement of existing features, or a new feature in existing plugins
    • I have updated CHANGELOG.next-release.md
    • I have added tests that prove my fix is effective or that my feature works
    • Added an API method or config option? Document in which version this will be introduced
    • I have made corresponding changes to the documentation
  • This is a bugfix
  • This is a new plugin
    • I have updated CHANGELOG.next-release.md
    • My code follows the style guidelines of this project
    • I have made corresponding changes to the documentation
    • I have added tests that prove my fix is effective or that my feature works
    • New and existing unit tests pass locally with my changes
    • I have updated supported-technologies.md
    • Added an API method or config option? Document in which version this will be introduced
    • Added an instrumentation plugin? Describe how you made sure that old, non-supported versions are not instrumented by accident.
  • This is something else

@github-actions

Copy link
Copy Markdown

🤖 GitHub comments

Just comment with:

  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown

Elastic Docs Style Checker (Vale)

Summary: 6 suggestions found

💡 Suggestions (6): Optional style improvements. Apply when helpful.
File Line Rule Message
docs/reference/config-core.md 332 Elastic.WordChoice Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI.
docs/reference/config-core.md 332 Elastic.WordChoice Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI.
docs/reference/config-core.md 356 Elastic.WordChoice Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI.
docs/reference/supported-technologies.md 136 Elastic.Semicolons Use semicolons sparingly. Consider splitting the sentence or using a comma or conjunction.
docs/reference/supported-technologies.md 136 Elastic.WordChoice Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI.
docs/reference/supported-technologies.md 136 Elastic.Semicolons Use semicolons sparingly. Consider splitting the sentence or using a comma or conjunction.

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

target.append("null");
return;
}
value = truncate(value, coreConfig.getLongFieldMaxLength());

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this makes truncating a single string value, however this is called more than once on a single field to reconstruct the whole json, so the effective value can be larger that the one configured.

return value.substring(0, value.offsetByCodePoints(0, maxCodePoints));
}

private static String serializeMessages(List<? extends Message> messages) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really have to reconstruct the whole json message here ? At the spring-ai level there should be no json involved at all, so here we re-create an artificial body of what the underlying implementation should send on the wire, is my understanding correct ?

@Advice.OnMethodEnter(suppress = Throwable.class, inline = false)
public static Object onEnter(@Advice.This ToolCallback callback,
@Advice.Argument(0) String toolInput) {
Span<?> span = SpringAiSpanHelper.startToolSpan(callback, toolInput);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we ever get nested spans due to wrapping, using CallDepth might help to avoid nested calls here and only instrument the outer call.


import javax.annotation.Nullable;

import static net.bytebuddy.matcher.ElementMatchers.*;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

avoid wildcard imports, use explicit ones instead.

public class SpringAiConfiguration extends ConfigurationOptionProvider {

private final ConfigurationOption<Boolean> captureAiContent = ConfigurationOption.booleanOption()
.key("capture_ai_content")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this configuration option is spring-specific, we should include 'spring' in its name as it's not used elsewhere.


import javax.annotation.Nullable;

import static net.bytebuddy.matcher.ElementMatchers.*;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

avoid wildcard imports.

Comment on lines +46 to +47
.and(classLoaderCanLoadClass(
"org.springframework.ai.vectorstore.observation.AbstractObservationVectorStore"));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why should we avoid instrumenting when this class is not present ? Is there any version where this class is not present and we need not to instrument or is it usually present ? If it's not present in the classpath the instrumentation won't be applied because it won't match.

this is a long way to say this is very probably not needed to override this method to do the extra check on this class name.

Comment on lines +20 to +22
<version.spring-ai-2-min>2.0.0</version.spring-ai-2-min>
<version.spring-ai-1-0>1.0.8</version.spring-ai-1-0>
<version.spring-ai-1-1>1.1.7</version.spring-ai-1-1>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing multiple multiple versions is usually done with dedicated tests similar to OpenTelemetryVersionIT This provides better isolation and avoid having messy dependencies in the pom.xml.

.hasName("chat gpt-test")
.hasType("app")
.hasSubType("internal")
.hasAction("chat")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is the 'action' still visible in the trace breakdown these days ? if not it's not worth to set nor test for it.

.hasType("app")
.hasSubType("internal")
.hasAction("chat")
.hasOutcome(co.elastic.apm.agent.tracer.Outcome.SUCCESS)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

avoid fqn names and use imports instead.

Comment on lines +122 to +136
EmbeddedSdkManager sdkManager = tracer.getLifecycleListener(EmbeddedSdkManager.class);
((SdkMeterProvider) sdkManager.getMeterProvider().getDelegate()).forceFlush().join(10, SECONDS);
assertThat(reporter.getBytes()).hasSize(1);
JsonNode metricSet = OBJECT_MAPPER.readTree(reporter.getBytes().get(0)).get("metricset");
JsonNode sample = metricSet.get("samples").get(SpringAiMetrics.TIME_TO_FIRST_CHUNK);
assertThat(sample.get("type").asText()).isEqualTo("histogram");
long count = 0;
for (JsonNode bucketCount : sample.get("counts")) {
count += bucketCount.asLong();
}
assertThat(count).isEqualTo(3);
assertThat(sample.get("values").size()).isGreaterThanOrEqualTo(2);
assertThat(metricSet.get("tags").toString())
.contains("gen_ai_operation_name", "chat", "gen_ai_request_model", "gpt-stream")
.doesNotContain("gen_ai_provider_name");

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we already have this information as a span attribute, do we also need to capture it as an histogram ? this is something that can be inferred by aggregating on the span traces document afterwards, there is no need to do this in the agent as it further complicates things (and the way to test it here is too complicated).

Comment on lines +219 to +220
.doesNotContainKeys("db.query.text", "db.vector.query.content", "db.vector.query.filter",
"db.vector.query.response.documents");

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any particular reason those keys should NOT be captured ? In the tests we usually test what should be present and not explicitly test for the things that are not expected.

This branch was successfully deployed

1 active deployment
docs-preview — 9591fb75 Deployed Nov 13, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants