Skip to content

fix(docker): silence expected probe connection failures for offline environments - #1752

Merged
jotka merged 1 commit into
Finsys:mainfrom
Totopo27:fix/docker-offline-log-flooding
Oct 15, 2026
Merged

jotka merged 1 commit into
Finsys:mainfrom
Totopo27:fix/docker-offline-log-flooding

Conversation

@Totopo27

Copy link
Copy Markdown
Contributor

Fixes #1739

Summary of Changes

  • Extracted pure helper functions isDockerProbePath and isDockerNetworkError into src/lib/server/docker-probe-core.ts.
  • In dockerFetch (src/lib/server/docker.ts), routed connection failures through logDockerFetchFailure.
  • Connection failures on lightweight probe endpoints (/_ping, /version, /_hawser/info) with expected network reachability errors (e.g. etch failed, ECONNREFUSED, ETIMEDOUT, EHOSTUNREACH) are downgraded to console.debug, preventing ERROR log flooding when remote hosts are powered off or offline.
  • Added comprehensive unit tests in ests/docker-offline-probe.test.ts.

…nvironments

Degrade fetch failures on lightweight probe and healthcheck endpoints
(/_ping, /version, /_hawser/info) to debug level when remote hosts are
offline or unreachable, preventing ERROR log flooding.

Fixes Finsys#1739
@CLAassistant

CLAassistant commented Oct 13, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@jotka
jotka merged commit da3916a into Finsys:main Oct 15, 2026
2 checks passed
@jotka

jotka commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Merged, thanks. Two follow-ups landed on top, one of them a mistake of mine.

The includes('timeout') check also matched failures that are not network ones - a
container stop timeout, a restic lock timeout, an exec timeout. Harmless on the three
probe paths, but the helper is exported, so I narrowed it to the errno codes plus
TimeoutError by name. The same predicate also already existed as a copied regex at two
call sites in docker.ts, so those now use the helper instead of carrying a third copy.

That narrowing was a regression and your version had it right: on HTTPS/mTLS and on the
edge relay the transport cuts the request short itself and throws a fresh Error, so the
name and the socket's errno are both gone - measured 5003ms, name=Error,
"Request aborted". Your includes('timeout') caught those; my errno check did not, so
unreachable mTLS and edge hosts kept logging on every cycle. Fixed by naming the error at
the seven places that synthesise one, so one predicate works on all four transports.

Also worth knowing: logDockerFetchFailure sat in docker.ts, which cannot be imported
under bun, so five mutations to it - including inverting the condition - passed the whole
suite. The decision moved into a pure function to make it testable.

One thing I left alone deliberately: ENOENT is not treated as unreachable, since it is
also what a missing socket path reports, and that is a configuration mistake which should
stay loud.

The log flooding is fixed, but the other half of #1739 is not: the Go collector still
probes an offline host every 30s with a retry, plus the disk check and the watchdog.
That needs a backoff and is a separate change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Excessive Docker API retries and ERROR log flooding for offline Hawser environments

3 participants