Repository navigation
Analyze OTLP spans opt-in by service.layer, and file the OTLP service catalog by layer - #14170
Conversation
… catalog by layer The trace analyzer's shipped rules (tal/otlp-general.yaml, formerly otlp-default.yaml) are declared for OTLP_GENERAL, renamed from OTLP_SERVICE, so only spans whose resource carries service.layer=OTLP_GENERAL are analyzed; the others are stored and searchable. OTLP_MESH is a new layer with no rules for service mesh sidecars. Only spans a rule claims name a service for host resolution and send pairing facts, and only spans some rule runs on take a dedup slot. The OTLP service catalog keeps each service's layer (otlp_service_layer; a registered layer, else UNDEFINED), one row per name and layer, and IOTLPTraceQueryDAO#getServices(layer) filters in storage. The stored span's service.layer carries the same value, so a TraceQL search on the layer agrees with the catalog, which answers a layer-only filter on the service name tags.
…-spans # Conflicts: # docs/en/changes/changes.md
There was a problem hiding this comment.
🟡 Changes recommended
Catalog identity collisions can lose service entries, and client pairing publication is not yet gated by a successful span claim.
2 open issues
What changed in this PR
This PR makes OTLP trace analysis opt-in through service.layer and carries that layer through SkyWalking’s native OTLP storage and TraceQL service catalog.
Changes:
- Renames the shipped TAL rules and layer to
OTLP_GENERAL, addsOTLP_MESH, and limits service registration and pairing to claimed spans. - Stores catalog entries by service and layer, normalizes stored resource layers, and adds storage-side filtering for TraceQL service pickers.
- Updates operator documentation, configuration defaults, unit tests, and end-to-end fixtures.
| File | Description |
|---|---|
| test/e2e-v2/cases/storage/expected/config-dump.yml | Expects the renamed default TAL file. |
| test/e2e-v2/cases/otlp-traces/native/tal-cases.yaml | Queries the OTLP_GENERAL layer. |
| test/e2e-v2/cases/otlp-traces/native/expected/tal-topology.yml | Updates expected topology layers. |
| test/e2e-v2/cases/otlp-traces/native/expected/tal-services.yml | Updates expected service layers. |
| test/e2e-v2/cases/otlp-traces/native/docker-compose-demo.yml | Opts demo services into analysis. |
| oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/dao/JDBCOTLPServiceCatalogTest.java | Tests JDBC layer filtering and deduplication. |
| oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/dao/JDBCOTLPTraceQueryDAO.java | Reads layer-aware JDBC catalog rows. |
| oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/otlp/OTLPTraceQueryEsDAO.java | Filters Elasticsearch catalog rows by layer. |
| oap-server/server-storage-plugin/storage-banyandb-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/trace/BanyanDBOTLPTraceQueryDAOTest.java | Adopts the updated catalog API. |
| oap-server/server-storage-plugin/storage-banyandb-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/trace/BanyanDBOTLPServiceCatalogTest.java | Tests BanyanDB layer-condition placement. |
| oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/trace/BanyanDBOTLPTraceQueryDAO.java | Reads and filters layer-aware catalog rows. |
| oap-server/server-starter/src/main/resources/tal/otlp-general.yaml | Restricts shipped rules to OTLP_GENERAL. |
| oap-server/server-starter/src/main/resources/application.yml | Changes the default rule file and deduplication guidance. |
| oap-server/server-receiver-plugin/otel-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/otel/otlp/ShippedRuleOverRecordedTraceTest.java | Loads the renamed shipped rule. |
| oap-server/server-receiver-plugin/otel-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/otel/otlp/OTLPStoredLayerHandlerTest.java | Tests stored versus analysis layer values. |
| oap-server/server-receiver-plugin/otel-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/otel/otlp/OTLPSpanForwardCatalogLayerTest.java | Tests resource normalization and catalog layers. |
| oap-server/server-receiver-plugin/otel-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/otel/otlp/OTLPSpanForward.java | Normalizes resource layers and populates catalog layers. |
| oap-server/server-receiver-plugin/otel-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/otel/otlp/OpenTelemetryTraceHandler.java | Separates received and stored resource layers. |
| oap-server/server-query-plugin/traceql-plugin/src/test/java/org/apache/skywalking/oap/query/traceql/handler/OTLPTraceQLServiceLayerTest.java | Tests catalog-backed layer filtering. |
| oap-server/server-query-plugin/traceql-plugin/src/main/java/org/apache/skywalking/oap/query/traceql/handler/TraceQLApiHandler.java | Shares filter classification helpers. |
| oap-server/server-query-plugin/traceql-plugin/src/main/java/org/apache/skywalking/oap/query/traceql/handler/OTLPTraceQLApiHandler.java | Answers layer-only service queries from the catalog. |
| oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/query/OTLPTraceQueryServiceTest.java | Tests service-name deduplication and filter forwarding. |
| oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/otlp/OTLPServiceTrafficTest.java | Tests layer identity, serialization, and defaults. |
| oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/query/IOTLPTraceQueryDAO.java | Defines the layer-aware catalog contract. |
| oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/OTLPTraceQueryService.java | Filters catalog reads and deduplicates names. |
| oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/otlp/source/OTLPService.java | Adds the catalog layer to service sources. |
| oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/otlp/OTLPServiceTraffic.java | Persists layer-specific catalog identities. |
| oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/otlp/dispatcher/OTLPServiceDispatcher.java | Propagates the source layer into traffic rows. |
| oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java | Renames layer 52 and registers OTLP_MESH. |
| oap-server/analyzer/trace-analyzer/src/test/java/org/apache/skywalking/oap/trace/analyzer/v2/ShippedRules.java | Points shared test helpers at the renamed file. |
| oap-server/analyzer/trace-analyzer/src/test/java/org/apache/skywalking/oap/trace/analyzer/v2/provider/TraceAnalyzerModuleProviderTest.java | Updates stored-service loading expectations. |
| oap-server/analyzer/trace-analyzer/src/test/java/org/apache/skywalking/oap/trace/analyzer/v2/provider/span/TraceAnalyzerServiceImplTest.java | Tests claimed-span registration and pairing. |
| oap-server/analyzer/trace-analyzer/src/test/java/org/apache/skywalking/oap/trace/analyzer/v2/provider/span/listener/SpanFilterListenerFactoryTest.java | Uses the renamed layer in rule validation. |
| oap-server/analyzer/trace-analyzer/src/test/java/org/apache/skywalking/oap/trace/analyzer/v2/provider/ShippedTalFileTest.java | Verifies shipped rules explicitly target OTLP_GENERAL. |
| oap-server/analyzer/trace-analyzer/src/test/java/org/apache/skywalking/oap/trace/analyzer/v2/dsl/TALWorkedExampleTest.java | Updates worked examples and expected entities. |
| oap-server/analyzer/trace-analyzer/src/test/java/org/apache/skywalking/oap/trace/analyzer/v2/dsl/TALExecutionTest.java | Updates runtime tests to renamed rules and layers. |
| oap-server/analyzer/trace-analyzer/src/test/java/org/apache/skywalking/oap/trace/analyzer/v2/compiler/TALSourceAttributionTest.java | Updates generated-source attribution expectations. |
| oap-server/analyzer/trace-analyzer/src/test/java/org/apache/skywalking/oap/trace/analyzer/v2/compiler/TALShippedRuleCompileTest.java | Compiles the renamed shipped rule. |
| oap-server/analyzer/trace-analyzer/src/test/java/org/apache/skywalking/oap/trace/analyzer/v2/compiler/TALScriptParserTest.java | Updates parser source-text expectations. |
| oap-server/analyzer/trace-analyzer/src/test/java/org/apache/skywalking/oap/trace/analyzer/v2/compiler/TALClassGeneratorTest.java | Uses the renamed layer in compiler fixtures. |
| oap-server/analyzer/trace-analyzer/src/test/java/org/apache/skywalking/oap/trace/analyzer/v2/compiler/DumpShippedRuleSourcesTest.java | Updates generated-file naming expectations. |
| oap-server/analyzer/trace-analyzer/src/main/java/org/apache/skywalking/oap/trace/analyzer/v2/provider/TraceAnalyzerModuleProvider.java | Loads known services from OTLP_GENERAL. |
| oap-server/analyzer/trace-analyzer/src/main/java/org/apache/skywalking/oap/trace/analyzer/v2/provider/TraceAnalyzerModuleConfig.java | Sets the renamed default TAL file. |
| oap-server/analyzer/trace-analyzer/src/main/java/org/apache/skywalking/oap/trace/analyzer/v2/provider/TALConfig.java | Updates the source-path documentation example. |
| oap-server/analyzer/trace-analyzer/src/main/java/org/apache/skywalking/oap/trace/analyzer/v2/provider/span/TraceAnalyzerServiceImpl.java | Gates deduplication, registration, and server pairing. |
| oap-server/analyzer/trace-analyzer/src/main/java/org/apache/skywalking/oap/trace/analyzer/v2/provider/span/listener/SpanFilterListener.java | Exposes whether auto-layer rules are installed. |
| oap-server/analyzer/trace-analyzer/src/main/antlr4/org/apache/skywalking/tal/rt/grammar/TALParser.g4 | Updates a comment example; grammar is unchanged. |
| docs/en/setup/backend/trace-analyzer.md | Documents opt-in analysis, mesh labeling, and deduplication. |
| docs/en/setup/backend/otlp-trace.md | Documents stored-layer normalization and opt-in behavior. |
| docs/en/setup/backend/configuration-vocabulary.md | Updates rule defaults and capacity guidance. |
| docs/en/concepts-and-designs/tal.md | Explains layer routing and claimed-span behavior. |
| docs/en/changes/changes.md | Records analysis and catalog changes. |
| docs/en/api/traceql-service.md | Documents layer-filtered service catalogs. |
Review effort: Balanced
A TAL rule is declared for a registered layer and sees only the spans that carry it; a span without a registered service.layer is stored, not analyzed. An auto rule picked the layer from the span, naming the service under a layer its stored spans and the OTLP service catalog did not carry, and would analyze unlabelled mesh sidecars. A rule declaring `auto`, or no layer, now stops the OAP from starting; a sender that cannot be changed gets service.layer from the collector instead.
wu-sheng
added
backend
Proposal: build TAL network aliases from strict span pairing only, keyed by caller service + peerThis is for the next update of this PR, which turns the client/server mapping into a TAL option. The goal is to stop deriving network aliases from name guessing and from Why the current sources are not reliable
Pairing works hop by hop: it matches a CLIENT span with its direct child SERVER span. On the same recorded trace it gives two aliases, whatever the
ProposalThe foundation, and a hard requirement: 1. Three TAL functions
2. The rule composes the alias address: caller service + peer The alias address is built by the TAL rule itself, on the server side, and the client side composes it the same way to look it up. The shipped rule uses:
3. PairingFact and the build
Both halves are routed to the trace owner by trace id, as they are now. On the owner:
If the composed address is longer than the 4. Lookups
5. Messaging stays 6. Removed
None of this has been released, so no compatibility path is needed, and the changelog note goes into the TAL entry. 7. Shipped rule (sketch) A database, cache or broker never answers with a SERVER span, so its peer is only looked up, never prepared. The slow-SQL rule (L317) looks it up the same way. 8. Costs to document
9. Extra key parts later. A rule that needs more than caller + peer, such as an attribute or a processed URL segment, changes the expressions of its 10. Docs. Rewrite the "Address-named peers" section of |
Follow-up: canary routed by an HTTP headerThis builds on the network alias proposal. The shipped rule stays as proposed. This case belongs on a best-practice page. The case
If The key: put the version into the peerOnly calls that carry the header get the suffix. Stable traffic keeps the shipped key What changes in TAL: only the CLIENT block of the rule
This is why the version belongs in the peer. Item 9 of the proposal still applies to a key part kept outside the peer: the fold cannot rebuild it. A part written into the peer is folded like any other. The whole change, against the shipped rule sketch in item 7 of the proposal: if (meta.kind == "SERVER") {
buildNetworkAlias callerScopedAddress(pairing.service, pairing.peer)
...
}
if (meta.kind == "CLIENT" || meta.kind == "PRODUCER") {
def peer = ...
+ // The canary header decides which service answers the call, so it is part of the alias key.
+ if (!isGateway(peer) && attr("http.request.header.x-canary") == "v2") {
+ def peer = peer + "(v2)"
+ }
if (meta.kind == "CLIENT" && !isGateway(peer)
&& !hasAttr("db.system.name", "db.system") && !hasAttr("messaging.system")) {
prepareNetworkAlias peer
}
if (!isGateway(peer)) {
def peer = resolveNetworkAlias(callerScopedAddress(meta.service, peer))
}
...
}The new Prerequisites and limits, for the doc
DocsThe shipped Outline:
|
…nd peer Network address aliases come only from the two spans of one call: a SERVER span whose parentSpanId is the CLIENT span's spanId. Each alias is keyed by the caller and the address it dialed, `frontend->cart:7070`, so one address dialed by two callers can name two services, and a proxy that keeps the Host header can no longer hand its address to the service behind it. - TAL: `prepareNetworkAlias` (CLIENT), `buildNetworkAlias` (SERVER, composed on the trace owner from `pairing.service` / `pairing.peer`), `callerScopedAddress` and `resolveNetworkAlias` replace the `alias` statement and `resolveAlias`; `KnownServices`, the host/DNS-label guard and the startup read of stored services are removed. - Pairing: two halves meet atomically on the trace owner; a learned pair is rebuilt once its row is ten minutes old by the OAP clock; at most one caller half per caller + peer every 30 s; `pairingSamplePercent` applies on a single node too; uncomposed and too-long pairs are counted and warned. - Topology: the fold tries the caller-scoped row before the plain address and keeps the row's own service id; a call to another replica is a loop. - Gateways: never aliased; a bare host is named `host:80` on both sides. - Docs: tal.md, trace-analyzer.md, otlp-trace.md and a new OpenTelemetry Trace Analysis Practices page.
Review of
|
Practices page: what is missingThis replaces 1.1 and 1.2 of my review comment. 1.2 is withdrawn: an uninstrumented gateway that splits one address by blue/green or weight is unsupported, as the page says. These practices exist to draw a correct topology: the gateway where it is, and each destination behind it on its own edge. The page covers the uninstrumented gateway in
TAL itself, the grammar, the runtime and the shipped rule, needs no change for any of these. |
…it changes - Drop the learned/reported/recentlySent caches: every sampled CLIENT and SERVER span sends its half and every matched pair pushes a NetworkAddressAliasSetup. - NetworkAddressAlias.combine skips the L2 write when the mapping is unchanged and was written less than five minutes ago, keeping rows inside the cache's ten-minute reload window; a changed mapping is written at the next flush. - Stamp last_update_time_bucket with the minute OAP receives the setup; the time bucket keeps the span's minute for retention. - Send the CLIENT half once after the rule loop, from the first non-aborted rule that named a peer. - Key OTLPServiceTraffic by Base64(service name) and layer. - Remove `layer: auto`; document gateways, canaries and gateway instrumentation.
nginx's OpenTelemetry modules report one SERVER span per request and inject its context upstream, so it is not a gateway that traces itself in the SpanKind sense. The practices page drops it from that table row, adds a row and a closing section on pairing both hops from its one span with a rule on its SERVER span, and the TAL doc says prepareNetworkAlias may run on such a span.
nginx's OpenTelemetry modules report one SERVER span per request and inject that span's own context upstream. SpanPairing.clientSeen now accepts a caller half from a SERVER span as well as from a CLIENT span, so a rule can send one from that span and both hops are learned. The shipped rule still prepares only in its CLIENT block. Docs and comments describe the pairing by caller and callee. The nginx config keeps only a single upstream address, since every retried attempt names nginx's span as its parent, and an nginx facing browsers needs its own sampling decision. A stale tal.md line saying prepareNetworkAlias checks the alias row is removed. Tests cover both hops from one span, a root proxy span, and the doc's rule compiled into a copy of the shipped one. The SpanPairingTest case asserting that a SERVER span sends no caller half drops that line and is renamed rootServerConsumerAndProducerSpansSendNoFact.
Pushed: nginx, a proxy that reports one span per request
nginx's OpenTelemetry modules, the official
The nginx behaviour, from source: one span per request, written at the log phase (http_module.cpp#L505-L521), always SERVER (batch_exporter.hpp#L35), its own context injected upstream (#L351, #L389); the contrib module the same (otel_ngx_module.cpp#L528, #L564). |
Final conclusion for mergingThe review is complete at
The UI side is apache/skywalking-horizon-ui#207. Merge once CI is green. The squash commit message: |
Analyze OTLP spans opt-in by
service.layer, and file the OTLP service catalog by layertrace-analyzer.md,tal.md,otlp-trace.md,traceql-service.md.TAL is opt-in by layer. The shipped rule file
tal/otlp-default.yamlis renamedtal/otlp-general.yaml, and its rulesotlp-generalandotlp-slow-sqlare declared forOTLP_GENERAL(renamed fromOTLP_SERVICE, ordinal 52). Only spans whose resource carriesservice.layer=OTLP_GENERALare analyzed; the others are stored and searchable. When nolayer: autorule is installed, the OAP logs one INFO line per service whose spans carry no layer.OTLP_MESH(56) is a new layer with no rules, for service mesh sidecars. Unlike LAL, TAL has nolayer: auto: a rule is declared for a registered layer and sees only the spans that carry it, and a rule declaringauto, or no layer, stops the OAP from starting. A sender that cannot be changed gets itsservice.layerfrom the OpenTelemetry Collector'sresourceprocessor instead.Claimed spans. Only a span a rule claims, one that ran without
abort {}, names its service for host resolution and sends a pairing fact; only a span some rule runs on takes a dedup slot.The OTLP service catalog keeps the layer.
OTLPServiceTrafficstores each service'sservice.layerinotlp_service_layer(a column name unique in the merged Elasticsearch metrics index): a registered layer, elseUNDEFINED. There is one row per name and layer.IOTLPTraceQueryDAO#getServiceNamesis replaced bygetServices(layer), which filters in storage on BanyanDB, Elasticsearch and JDBC. The stored OTLP span's resourceservice.layeris set to the same value, so a TraceQL search on the layer agrees with the catalog, while TAL reads the layer as sent.TraceQL. A
qwhose only condition is the scopedresource.service.layeris answered from the catalog forresource.service.name,resource.serviceandservice.name.Tests. New unit tests cover the catalog row, the three DAOs' layer filter, the TraceQL layer filter and the stored layer. The renamed tests and e2e files are updated: the native otlp-traces compose sets
service.layer=OTLP_GENERAL, and the config dump expectsotlp-general.Verified on kind with the showcase: the OpenTelemetry Astronomy Shop under
OTLP_GENERAL, Istio sidecars exporting OTLP underOTLP_MESH, and unlabelled and misspelled senders underUNDEFINED.CHANGESlog.