feat(ingest): open the front door — ingest key, generated contract, setup prompt, live status

.github/workflows/ci.yml

@@ -838,6 +838,10 @@ jobs:
             tests/test_otel_export_sessions_shape.py \
             tests/test_query_contract_drift.py \
             tests/test_public_api_keys.py \
+            tests/test_ingest_key.py \
+            tests/test_ingest_contract_drift.py \
+            tests/test_setup_prompt.py \
+            tests/test_ingest_status.py \
             tests/test_query_contract_goldens.py \
             tests/test_local_store_concurrent_flush_1590.py \
             tests/test_duckdb_invalidated_recovery.py \

CHANGELOG.md

@@ -522,6 +522,39 @@
 - **Cloud is unaffected by construction:** the hosted container has no discovery file, so no round trip is attempted and nothing is stamped.
 - **Verified:** 17 tests in `tests/test_store_unreachable_is_not_empty.py`, named in `ci.yml`, each proven to fail against the unfixed code before being trusted. Closes #5534.
 
+### Added: the first-run gate now says whether data is actually arriving (2026-09-08)
+- **Why:** a user who installs ClawMetry and sees an empty dashboard cannot tell "nothing is running" from "it is broken", and that question is what kills setup funnels. We have the number: 285 launches produced 13 choices in 14 days on the old gate. The gate already named the runtimes it detected; it never said whether any of them had produced a single event.
+- **What:** `GET /api/onboarding/ingest-status` answers it from real data: total events, the runtimes actually sending, and the OTLP receiver's own state; and the gate renders it as a live strip that polls while it is open. When nothing has arrived the strip says so **and what to do about it**, including `clawmetry setup-prompt` for an agent that runs somewhere else. It never blocks: this is a confirmation, not a step, in a flow whose whole selling point is having no steps.
+- **Two clocks, kept apart on purpose.** The durable count comes from DuckDB and survives a restart; the OTLP receiver's counters are in-process and empty on restart, so they are reported separately and named `has_data_this_process`. Folding them together would tell a working install it was broken every time the dashboard restarted.
+- **A runtime we merely know about is not a runtime that is sending.** Rows with no sessions and no tokens are dropped rather than listed, because listing them would answer "is anything arriving?" with a yes they have not earned, the same shape as a tab that renders empty and calls it success. And the rollup is one row per runtime *per day*, so a runtime sending for a week arrived seven times; the endpoint collapses to one row each.
+- **Read through `local_query._dispatch`, never from raw files**, so it answers identically on a laptop and in a cloud container with no `~/.openclaw`. Memoised for 2s because it is polled: the first call is ~200 ms against a real store, the rest are ~0.2 ms.
+- **Verified in both states against a running dashboard:** the strip renders inside the card in the real page, an empty store returns `connected: false` with an actionable `next_step`, and the populated store on this machine returns 25,349 events across 9 runtimes. 9 guards in `tests/test_ingest_status.py`, three mutation-proven: reporting an idle runtime as a source, dropping the memo, and never stopping the poll each turn one red. The last one exists because a dismissed modal that keeps fetching is the same defect as the Home widget that fetched every sub-agent into a hidden element.
+- **Refs** #5680.
+
+### Added: a setup prompt you hand to your agent (2026-09-08)
+- **Why:** ClawMetry's users delegate work to coding agents by definition (that is what the product observes), yet the only setup paths shipped were "run the installer" and "read a doc", neither aimed at the thing the user actually drives. `clawmetry setup-prompt <runtime>` and `GET /api/setup-prompt` print a prompt written for the agent, for the case auto-detection cannot cover: an agent in CI, a container, a serverless function, or on somebody else's laptop.
+- **Generated, not written.** Every endpoint, header, content type and cap comes from `clawmetry/ingest_contract.py`, the same declaration the server validates against. A hand-written prompt drifts the first time a header is renamed, and a drifted setup prompt is worse than none: the agent writes the wrong header *confidently* and the request fails where nobody is looking.
+- **Half of it is negative space**, which is the useful half. Coding agents reliably mis-substitute secrets, "correct" a content type that was already right, and invent config keys that look plausible. So the prompt says the key goes in one header and nowhere else; that the placeholder is a placeholder and the real key must be asked for, not invented; that both encodings are already accepted and need no fixing; that a key not in the prompt does not exist; and that this is observability: it watches, it does not change what runs. It ends by making the agent verify and report the real event count, because an agent that checks its own work fails loudly instead of silently.
+- **Two guards worth keeping.** The first reads *backwards*: every `x-clawmetry-*` token in the prompt must be a header the contract declares. Checking only that the right headers appear was too weak, proven by mutation, where swapping the config block's header for `x-clawmetry-apikey` left every other assertion green because the correct name still appeared in the prose. The second is general: **every registered subcommand must also be in `cli.py`'s `_subcmds` allowlist**, because a parser without an entry there falls through to the dashboard's argparse and dies with "invalid choice", which reads like the command was never written. `setup-prompt` did exactly that when first added (the same two-list trap CLAUDE.md documents for runtimes), and asserting only that `setup-prompt` is present would not have prevented the next one.
+- **The seam holds:** this module names no runtime and hardcodes no vendor value, pinned by a test. Where a runtime has registered an OTel profile (paid runtimes register theirs from clawmetry-pro), its label and `clawmetry instrument` support are read at render time; a free install gets the generic OTLP prompt, which works.
+- **Refs** #5681.
+
+### Added: the ingest contract is declared once, and the reference is generated from it (2026-09-08)
+- **Why:** four things describe what ClawMetry accepts: the server that validates requests, `docs/INGEST.md`, the per-runtime setup prompts, and the public reference on the landing site. None of them shared a source. Four hand-maintained descriptions of one contract is four chances to drift, and the drift is worst in the prompts: a prompt that teaches an agent a header we do not accept is worse than shipping no prompt at all, because the agent writes it confidently and the failure is silent. The landing side now enforces this from the other direction too: drift-bot fails a public claim the repo denies.
+- **What:** `clawmetry/ingest_contract.py` declares the surfaces, headers, encodings, caps, response codes, auth modes and GenAI attributes as data. `clawmetry/ingest_auth.py` imports its constants from there rather than keeping its own, so the thing the server enforces and the thing we publish cannot diverge. `scripts/gen_ingest_doc.py` renders `docs/INGEST.md` from it with a `--check` mode, on the same pattern as `gen_query_contract_doc.py` on the read side.
+- **The guard that earns the file.** Declaring "we read `gen_ai.usage.cache_read.input_tokens`" is cheap; the claim is only worth printing if something checks it. The test asserts every attribute declared read is actually named in the mapper **and** that every attribute declared unread really is, in both directions, because someone wiring one up should have to move it in the same change, so the published reference is never behind the code either. Writing that check is what surfaced #5685: we were advertising a convention we did not implement, and cached tokens were being priced as free.
+- **The doc says what we do NOT accept**, and that section is pinned by a test. There is no endpoint for syslog, CEF, GELF or raw text, and there is not going to be: ClawMetry's inputs are typed on arrival, and a parser layer would exist only to accept data this product has nothing to say about. A reference that only says what works is not one anybody can plan against.
+- **Verified by mutation:** claiming a made-up attribute is read turns 2 guards red; claiming a genuinely-read one is unread turns 3 red. 26 guards in `tests/test_ingest_contract_drift.py`, registered in `ci.yml`.
+- **Refs** #5682.
+
+### Added: an ingest key, so an agent that is not on this machine can be observed (2026-09-08)
+- **Why:** until now an agent was observable by ClawMetry only if the daemon ran on the same machine as the agent. `/v1/{logs,metrics,traces}` trusted loopback and otherwise wanted the OpenClaw gateway token; the custom-runtime write API trusted loopback or one static `CLAWMETRY_INGEST_TOKEN` shared by the whole install: no rotation, no revocation, no way to tell two pushers apart. Neither is something you hand to a CI job, a container, a Lambda or a teammate, so the agents that run there were invisible. That is one capability, not a catalogue, and it is what stands between the product and "every agent, anywhere it runs".
+- **What:** `write:ingest`, a scoped key created with `clawmetry key create --name ci --scope write:ingest` and presented as `x-clawmetry-key: cmk_...` on the three OTLP endpoints. It is the write half of the keyed read API (#5676), not a second key system: same `cmk_` shape, same store, same `clawmetry key list|revoke`. Plus two routing headers (`x-clawmetry-runtime` and `x-clawmetry-env`) because a pushed batch carries no filesystem layout to infer a runtime from. They are written into the resource attributes the mappers already read (`service.name`, `deployment.environment`), so a header is exactly as powerful as the equivalent exporter setting and no mapper learns a second way to answer the same question. Deliberately one grouping axis, not a dataset/collection/tag taxonomy: that is what a log platform needs and an agent platform does not.
+- **The posture, stated plainly.** An ingest key **can only push**: `write:ingest` grants no `q/1` shape, so a key handed to a CI runner cannot read a prompt, a cost or a session back out, and presenting one to `/api/q/1` is a `403` rather than an index it could never follow up on. It is **never given a CORS header** and `apikeys.create` refuses to put a browser origin on one, so a page cannot hold one usefully: a write surface is not the place to hand back the protection the read API was careful to keep. Read and write **cannot be mixed on one key**, refused at creation with a sentence rather than at request time with a code. And there is **one gate, not two**: `_check_auth` steps aside for a keyed `/v1/` request exactly as it does for `/api/q/`, so `clawmetry/ingest_auth.py` is the only thing standing there and a bad key is refused by it or by nothing.
+- **Every refusal carries a sentence.** These are read inside an agent's terminal output with no documentation open, so `401` says how to create a key, `403` says the key is fine but may not push, `400` on a bad runtime header shows the shape it wanted, `400` on a bad body names both accepted encodings, and `413` gives the size and the limit instead of failing somewhere inside a protobuf parser.
+- **Verified live, not only in tests.** Against a real dashboard on port 8917 with a gateway token set: all three doors return 200 (loopback with no key at all, the zero-config path, unchanged; gateway token; ingest key), and every refusal returns its own status with its own sentence. A span pushed with `x-clawmetry-runtime: my-engine` lands in DuckDB as `agent_type=my_engine, service_name=my-engine`, while the same span pushed without headers still lands as `openclaw`/`unknown_service`, the pre-existing behaviour, untouched. The live run is also what found the `/api/q/1` hole: a write-only key was being handed a 200 index, now a 403, pinned by a test. 25 guards in `tests/test_ingest_key.py`.
+- **Refs** #5679. Part of phase 1 of the ingest plan (#5680, #5681, #5682, clawmetry-cloud#2343, #2344, clawmetry-pro#230).
+
 ### Fixed: two functions named `_session_cwd`, and the later one silently replaced the other (2026-09-08)
 - **Why:** shipped in 0.12.837 and live through 0.12.839. `sync.py` already had `_session_cwd(row)`, a raw adapter/gateway dict read through a **twelve-alias** set (`cwd`, `workingDir`, `workingDirectory`, `workspace`, `workspaceRoot`, `project_dir`, `projectRoot`, `directory`, `folder`, ...). The workspace-scan change added a second function with the same name 2,000 lines below it, reading two keys. Python keeps the last definition and says nothing, so all THREE callers of the original quietly switched: gateway session shaping (`sync.py:4328`), the session row build (`4586`), and the FAMILY ingest cwd (`15293`). That last one writes `sessions.cwd`, which is the column `process_control` promotes to find a pid and the one the workspace scanner keys on, so a runtime that spells its directory `workingDir` or `directory` in metadata stopped persisting it. Nothing failed. The tests passed. Three wheels shipped.
 - **What:** the newer helper becomes `_session_row_cwd` (it reads a store ROW: the `cwd` column first, then metadata), and its metadata fallback now DELEGATES to `_session_cwd`, so both paths honour the same alias set. Both are strictly better than before the collision. A second shadowing found by the same walk is also removed: `start_log_streamer` was defined twice, the first an empty docstring-only stub.

clawmetry/apikeys.py

@@ -34,6 +34,14 @@
   is created with ``--origin none`` and simply never gets a CORS header.
 * Scopes are least-revealing-first and ``read:content`` is never
   granted implicitly -- it has to be asked for by name.
+* Read scopes are read-only. ``routes/public_api.py`` dispatches only
+  ``q/1`` read shapes, so a browser-resident key adds nothing to
+  ClawMetry's control plane.
+* There is exactly one write scope, ``write:ingest``, and it can only
+  push telemetry IN. It cannot read a single byte back, and nothing in
+  this module can pause, stop or kill an agent. An ingest key is
+  server-to-server: it is never granted a CORS header, so a page cannot
+  hold one usefully (see ``ingest_auth``).
 * Keys are read-only. Nothing in this module can pause, stop or kill an
   agent, and ``routes/public_api.py`` dispatches only ``q/1`` read
   shapes, so this adds nothing to ClawMetry's control plane.
@@ -57,6 +65,11 @@
 import time
 from typing import Optional
 
+from clawmetry.query_contract import (
+    SCOPE_CONTENT,
+    SCOPE_DOC as _READ_SCOPE_DOC,
+    SCOPES as READ_SCOPES,
+)
 from clawmetry.query_contract import SCOPES
 
 # Re-export the read-side public helpers from their own short module so
@@ -92,6 +105,23 @@
 #: script cannot grow the file without bound; it is not a paywall.
 MAX_KEYS = 50
 
+#: The one write scope. It lives here rather than in ``query_contract``
+#: on purpose: that module declares what can be READ, shape by shape,
+#: and a scope with no shape behind it would be a lie in that table.
+#: Ingest is the opposite direction and has no q/1 method at all.
+SCOPE_INGEST = "write:ingest"
+
+#: Every scope a key may carry. Read scopes stay in their declared
+#: least-revealing-first order; the write scope sorts last because it is
+#: the one a reader should notice.
+SCOPES: tuple = tuple(READ_SCOPES) + (SCOPE_INGEST,)
+
+SCOPE_DOC: dict = dict(_READ_SCOPE_DOC)
+SCOPE_DOC[SCOPE_INGEST] = (
+    "Push telemetry in: OTLP logs, metrics and traces, and run events. "
+    "Grants no read access of any kind."
+)
+
 #: Sentinel origin meaning "this key is not used from a browser". Stored
 #: as an empty origin list; kept as a word so the CLI can say it back.
 ORIGIN_NONE = "none"
@@ -307,6 +337,25 @@ def create(name: str, scopes, origins, *, note: str = "") -> tuple:
     scope_list = normalise_scopes(scopes)
     origin_list = normalise_origins(origins)
 
+    # An ingest key is server-to-server and is never granted a CORS
+    # header, so browser origins on one would be dead configuration that
+    # reads like a permission. Refusing the mix also keeps a single key
+    # from being both "pasted into a web page" and "allowed to write",
+    # which is the combination worth not having.
+    if SCOPE_INGEST in scope_list:
+        if len(scope_list) > 1:
+            raise ApiKeyError(
+                "An ingest key does one job. Create it with write:ingest "
+                "alone, and mint a separate read key for anything that "
+                "needs to read data back."
+            )
+        if origin_list:
+            raise ApiKeyError(
+                "An ingest key is used by a server, a container or a CI "
+                "job, never by a browser, so it takes no origin. Create "
+                "it with --origin none."
+            )
+
     doc = _read_store()
     live = [k for k in doc["keys"] if not k.get("revoked_at")]
     if len(live) >= MAX_KEYS:
@@ -560,6 +609,12 @@ def any_canonical_allowed_origin(origin: str) -> "str | None":
     return None
 
 
+def allows_ingest(record: dict) -> bool:
+    """True when this key may push telemetry in."""
+    return SCOPE_INGEST in (record.get("scopes") or [])
+
+
+
 def redact(presented: str) -> str:
     """``cmk_a1b2c3d4_...`` -- safe to log. Shows the id, never the secret."""
     parsed = parse(presented)

clawmetry/apikeys_public.py

@@ -39,27 +39,37 @@ def granted_shapes(record: dict) -> set:
 
 
 def scope_catalogue() -> list:
-    """``[{scope, doc, methods, sensitive}]`` for the UI and the CLI help.
+    """``[{scope, kind, doc, methods, sensitive}]`` for the UI and the CLI help.
 
-    Derived from the query contract, so a method added there shows up
-    here with no second list to update.
+    Read scopes are derived from the query contract; write:ingest is loaded
+    lazily from apikeys so the ingest scope doc stays in one place.
     """
     from clawmetry.query_contract import (
         SCOPE_CONTENT,
         SCOPE_DOC,
         SCOPES,
         live_methods_by_scope,
     )
+    from clawmetry.apikeys import SCOPE_INGEST, SCOPE_DOC as _AK_SCOPE_DOC
 
-    return [
+    rows = [
         {
             "scope": s,
+            "kind": "read",
             "doc": SCOPE_DOC[s],
             "methods": live_methods_by_scope(s),
             "sensitive": s == SCOPE_CONTENT,
         }
         for s in SCOPES
     ]
+    rows.append({
+        "scope": SCOPE_INGEST,
+        "kind": "write",
+        "doc": _AK_SCOPE_DOC.get(SCOPE_INGEST, ""),
+        "methods": [],
+        "sensitive": False,
+    })
+    return rows
 
 
 def store_summary() -> dict[str, Any]: