fix(0281): make ClickHouse errors readable — they were all empty

[karczuRF]

Summary

• Every ClickHouse error reaching production was BadResponse("") — an empty string. Not only timeouts: unknown table, syntax error, quota exceeded and disk-full all arrived with the code and message stripped off, indistinguishable from a network blip.
• The cause is a fallback that never fires, plus the crate's default compression. Fixed by routing every client this crate builds through a new with_readable_errors.
• New integration test, plus scripts/ch-proxy-0281.sh to stand up the proxy it needs.

How it was found

0215's acceptance criterion said "verified by inducing, not inferred". The induction on 2026-09-11 failed: the bound fired exactly (ClickHouse killed the statement at 1002.3 ms against 1000 ms) but the worker logged clickhouse: bad response: . Everything needed to diagnose the failure existed on the wire and was lost one layer above it.

The mechanism

collect_bad_response LZ4-decodes the error body and falls back to the raw bytes on failure:

let bytes = collect_bytes(stream).await.unwrap_or(raw_bytes);

Straight to ClickHouse the decode fails, the fallback fires, the message survives. Through a proxy the chunk reframing makes the decode succeed with zero bytes — Ok(empty) — so unwrap_or never runs. Every production client reaches ClickHouse through Caddy, and our client never set compression, so it ran the crate default of Lz4.

Measured against CH 26.3.10.60, the same statement three ways:

path	error the caller receives
direct to ClickHouse	bad response: Code: 159. DB::Exception: Timeout exceeded: elapsed 1000.178179 ms, maximum: 1000 ms. (TIMEOUT_EXCEEDED)
through Caddy	bad response:
through Caddy, compression off	the full message returns

curl sees the body in all three — it's specific to the crate's decode path.

Why the fix lives in a named function

One setting, but placed in with_readable_errors rather than inline, so the test exercises the same code production does. mtls::client_with_mtls — the single funnel every worker, probe, API and operator CLI goes through — calls it.

The test fails when its proxy is missing, deliberately

Straight to ClickHouse this test passes whether or not the defect is present, because the fallback fires. So CLICKHOUSE_PROXY_URL being unset is an error, not a skip — a test that cannot fail is worse than no test, because it reports success.

scripts/ch-proxy-0281.sh up
CLICKHOUSE_PROXY_URL=http://localhost:8124 \
  cargo test -p prices-clickhouse --test execution_bound_error_it -- --ignored

Verified red before the fix, with the exact message the error carries no message at all — this is the 0281 defect: "bad response: ", and green after.

Trade-off, stated

Response bandwidth on reads, since ClickHouse no longer compresses result bodies. Worth paying against errors that cannot be diagnosed at all. Revisit if the crate fixes the fallback upstream — we are pinned to 0.13.3 and 0.15.x is unexamined.

Closes

0215's last acceptance criterion depends on this. Once deployed, the induction can be re-run and should produce TIMEOUT_EXCEEDED (159) instead of an empty body.