feat(otel): LangGraph + OpenLLMetry recipe, CI-tested against a live ClawMetry (a run is no longer split in two) - #5972
Conversation
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
Visual diffComparing 42 of 70 comparison(s) flagged (>1% pixel diff).
Folder: 887537a7c45e. Full PNGs also attached as a workflow artefact. Generated by visual-diff bot. Pixel diffs >1% flagged; eyeball the table before merging. This check is non-blocking — fail = bot bug, not a code problem. |
|
The first run of the new Cause: readiness was probed on Reproduced locally in a venv installed from Fix (f4c7641): readiness now probes Rehearsed the job's steps locally before pushing: CI-pinned dashboard on |
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
Coordinator reviewVerdict: fix first, one blocking item. Everything else checks out. Reviewed at Blocking1. AC-OBS-OTR-001.4 says the opposite of what the PR ships. Factory REQ-OBS-OTR-001 and its copy in
The PR does the reverse in three places:
This is not only about the resource The product record and its guard disagree, and Drift Bot did not catch it. Pick one:
Verified (not taken from the builder report)
Non-blocking (worth doing in this PR or a follow-up)
|
…cedence Review on #5972 found the product record and its guard disagreed: AC .4 said a sent session identifier outranks the thread, while _otel_to_row, the page and the test rank the thread above session.id. The code is the right side of that: if a span-level session.id beat the thread, an app that stamps session.id on every span splits the run again (top span via gen_ai.conversation.id, children via session.id). - AC .4 reworded (tracked edit on the Factory requirement, mirrored in docs/acceptance_criteria.json): conversation id > thread > session.id on the span or resource, the winner recorded as sent. - The test now covers session.id on the span as well as the resource; both cases fail if session.id is read before the thread key. - The code comment no longer says only a resource session.id is outranked. - ingest_contract.py lists the thread key among the conversation-id fallbacks; docs/INGEST.md regenerated. - The page states the rule and why, notes the dashboard serves plain HTTP (a TLS terminator is needed for the https:// example), and says the tab table was checked through API endpoints, not rendered tabs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jm9d7s4fN55hN3YzQo75o9
|
Review follow-up (the blocking item on AC-OBS-OTR-001.4), fixed in 44d0eb8: Decision: keep the behaviour, correct the record (option a). The precedence is: sent
Non-blocking items taken:
Not taken: the bare-thread-id collision across 🤖 Generated with Claude Code |
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
|
Ready to merge (at 44d0eb8). Every check passes except OpenSSF Scorecard, which is skipped. That includes E2E Gate (required), Drift Bot, the product-record gate, CodeQL and the Merge-after dependencies: none. The branch is on current Companion PRs: none. There is no new HTTP route, so no cloud route-policy entry is needed. Before or at merge: accept the pending tracked edit on AC .4 of REQ-OBS-OTR-001 in Factory: https://factory.8090.ai/project/b415065f-ab2f-4f53-8864-0c009fd098cb/requirements/0130f6a6-3f5b-448f-bd59-d8820a4d5c88. After merge (on
After release: repeat the CI job by hand at the release tag, using a scratch HOME and scratch venvs, never the real
Still open on #5939:
🤖 Generated with Claude Code |
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
…try in CI Refs #5939. REQ-OBS-OTR-001. Running a real one-tool LangGraph agent (langgraph 1.2.11, opentelemetry-instrumentation-langchain 0.62.3) against the receiver found that a run was split in two: the instrumentation sends the thread as gen_ai.conversation.id on the top invoke_agent span only, and as traceloop.association.properties.thread_id on every span beneath it. The receiver read only the former, so the thread's session had 0 tokens and a per-trace session held every token and the tool call. _otel_to_row now reads the thread association right after the conversation id, recorded as sent. - examples/otel/langgraph/: the recipe (one tool, stub model, no key) - scripts/verify_otel_recipe_langgraph.py + ci.yml otel-recipe-langgraph: hash-pinned recipe venv, dashboard on a non-loopback address, checks 401 without Bearer, one session for invoke + stream on a thread, exact tokens, tool span under invoke_agent, per-trace session with no thread - tests/test_otel_recipe_langgraph.py: real captured exports through the receiver (4 red on main) - docs/OTEL_RECIPE_LANGGRAPH.md: versions, encoding, auth, structure, identity, per-tab behaviour, observe-only limits Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jm9d7s4fN55hN3YzQo75o9
The first CI run of otel-recipe-langgraph failed at boot: on a fresh store path the DuckDB file does not exist until the receiver's first write, so the read-only local API answers 503 "database does not exist" and the readiness loop never passed. Reproduced locally in a venv installed from ci-tests.txt + ci-otel-extra.txt; /api/health answered 200 in 3s. Readiness now uses /api/health like every other job, and the dashboard runs with -u so a boot failure prints its log. Rehearsed the whole job locally: verifier PASS including the non-loopback 401/Bearer step. Refs #5939 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jm9d7s4fN55hN3YzQo75o9
…cedence Review on #5972 found the product record and its guard disagreed: AC .4 said a sent session identifier outranks the thread, while _otel_to_row, the page and the test rank the thread above session.id. The code is the right side of that: if a span-level session.id beat the thread, an app that stamps session.id on every span splits the run again (top span via gen_ai.conversation.id, children via session.id). - AC .4 reworded (tracked edit on the Factory requirement, mirrored in docs/acceptance_criteria.json): conversation id > thread > session.id on the span or resource, the winner recorded as sent. - The test now covers session.id on the span as well as the resource; both cases fail if session.id is read before the thread key. - The code comment no longer says only a resource session.id is outranked. - ingest_contract.py lists the thread key among the conversation-id fallbacks; docs/INGEST.md regenerated. - The page states the rule and why, notes the dashboard serves plain HTTP (a TLS terminator is needed for the https:// example), and says the tab table was checked through API endpoints, not rendered tabs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jm9d7s4fN55hN3YzQo75o9
The test references this criterion for the recipe-page verification requirement but it was missing from the manifest, failing the AC traceability gate. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KUtV6jyUVMhBRWXjSSef9S
fcf68bd to
a223c24
Compare
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jm9d7s4fN55hN3YzQo75o9
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
… Factory Syntax & Lint failed: manifest has duplicate criterion ids. The later entry matches REQ-OBS-OTR-001 in 8090 Software Factory verbatim. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jm9d7s4fN55hN3YzQo75o9
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
Resolves conflicts in: - .github/workflows/ci.yml: keep both otel-recipe-langgraph job (this branch) and selfhosted-server job (main) at same insertion point; each gets its own steps block - docs/acceptance_criteria.json: keep both OTR-001 LangGraph recipe criteria and SHI-001-004 self-hosted image criteria from main Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B5EycwbWKDtGZtoAHVdDSA
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
2 participants
Refs #5939
Requirement: https://factory.8090.ai/project/b415065f-ab2f-4f53-8864-0c009fd098cb/requirements/0130f6a6-3f5b-448f-bd59-d8820a4d5c88 (REQ-OBS-OTR-001, child of Runtime and Session Observability; written before the code, blueprint filled in)
Branched off
main. Independent of #5953 (spans reach Guard) and #5963 (durable ack): either merge order works. The Guard line of the issue stays with #5953.Scope: the pilot-first increment
The issue's refinement asks for one real framework and exporter version first, CI-tested end to end, with other frameworks added only as separately validated increments. This PR is that first increment: LangGraph + OpenLLMetry. Azure AI Foundry and AWS AgentCore are not included and are listed below.
What running the real thing found
A one-tool LangGraph agent (langgraph 1.2.11, opentelemetry-instrumentation-langchain 0.62.3, opentelemetry-sdk 1.44.0) exported to a scratch dashboard came back as two sessions for one run:
t-1(the thread)proto:trace:<trace_id>lookup_invoiceOpenLLMetry sends the thread as
gen_ai.conversation.idon the topinvoke_agentspan only. Every span beneath it (both model calls and the tool call) carries it astraceloop.association.properties.thread_id._otel_to_rowread only the first, so the children fell through to the per-trace fallback.Fix (one pick list in
dashboard.py::_otel_to_row): readtraceloop.association.properties.thread_idright aftergen_ai.conversation.idand beforesession.id, recorded as sent. Precedence: sent conversation id, thread,session.id, then<app>:trace:<trace_id>. It is not prefixed, because the top span already sends the same value bare; a prefix would recreate the split.What ships
--thread,--streamexamples/otel/langgraph//api/local/traces+/api/local/spansscripts/verify_otel_recipe_langgraph.pyotel-recipe-langgraph: recipe in its own venv from a hash-pinned lockfile, dashboard on0.0.0.0, agent exports through the runner's non-loopback IP.github/workflows/ci.yml,.github/requirements/otel-recipe-langgraph.txttests/test_otel_recipe_langgraph.py,tests/fixtures/otel/docs/OTEL_RECIPE_LANGGRAPH.md(linked fromOPENTELEMETRY.md,BRING_YOUR_OWN_AGENT.md)The verifier checks, each a hard failure with a sentence:
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer …is accepted;invoke()thenstream()) are one session holding both traces;lookup_invoicespan descends frominvoke_agentin the same trace; both model calls carry the model and a derived cost; all spans are attributed toinvoice_agent;invoice_agent:trace:<trace_id>.Verified (locally, Python 3.11, scratch HOME and store, random port)
main(dashboard.pyswapped fororigin/main's copy): 4 hermetic tests fail (thread_run_is_one_session_with_every_span,second_run_on_the_thread_joins_the_same_session,streamed_run_matches_invoked_run,thread_wins_over_resource_session_id), and the live verifier fails withtrace …: tokens 0/0, the model reported 280/27.PASSincluding the 401/Bearer step through the machine's LAN address.test_otel_recipe_langgraph,test_otlp_session_materialization,test_otlp_sessionless_spans,test_spans_otlp_edge_cases,test_otlp_runtime_profiles,test_otlp_traces_cost,test_otlp_json_no_protobuf.check_ac_coverage --check(8 criteria mirrored, 117/187),check_ci_test_coverage --check(baseline tightened 920 to 919),gen_module_map --check,check_py39_annotations, ruff on the new files.cloud_route_policyentry is needed.Found while verifying, recorded on the page, not fixed here
openclaw, offers Pause/Stop, and suggestsopenclaw attach. None of that can reach a remote agent. The page says so and does not claim control. It belongs with the spans and Guard work (OTLP traces: spans bypass Guard detectors and redaction #5938).Remaining for #5939 (not in this PR)
examples/otel/azure-ai-foundry/andexamples/otel/aws-agentcore/, each with its own CI-validated recipe and page.🤖 Generated with Claude Code
https://claude.ai/code/session_01Jm9d7s4fN55hN3YzQo75o9