Skip to content

Avoid task replacement when registry digest resolution fails - #140

Open
Cicibot2026 wants to merge 1 commit into
containrrr:masterfrom
Cicibot2026:fix/avoid-unresolved-image-updates
Open

Cicibot2026 wants to merge 1 commit into
containrrr:masterfrom
Cicibot2026:fix/avoid-unresolved-image-updates

Conversation

@Cicibot2026

Copy link
Copy Markdown

A transient registry failure during docker service update --image=<tag> can remove an existing digest and replace healthy tasks with the same image. A subsequent successful resolution adds the digest back and causes another replacement. This is the tag/digest flapping reported in #110.

The existing docker manifest inspect preflight does not prevent this: the registry can become unavailable between that check and Docker's subsequent tag resolution.

This change resolves the target digest with Buildx before modifying the service, rejects failed or invalid resolutions, and passes the resolved reference to the update command. An unchanged reference is skipped unless UPDATE_OPTIONS is set, preserving explicitly requested update options such as --force. Each later cycle still resolves the original tag, so this does not freeze updates at a particular version.

Compatibility:

  • Uses Buildx already included in the existing docker:28 base image; no additional packages.
  • Retains the existing per-service Docker config and registry authentication options.
  • Keeps explicit WITH_NO_RESOLVE_IMAGE and WITH_INSECURE_REGISTRY modes on their existing path. This PR does not claim to fix digest flapping in those opt-in modes; Buildx has no equivalent to the current manifest inspect --insecure flag.
  • Uses the top-level registry digest, including an OCI index, rather than selecting a platform-specific manifest.

Validation:

  • bash -n shepherd, the existing shellcheck --shell bash shepherd, and a build using the unchanged Dockerfile all pass. The built image contains Docker 28.5.2 and Buildx 0.29.1, with the expected script content.
  • Eleven temporary command-path checks cover unchanged/new references, legacy unqualified references, failed preflight/resolution, malformed digests, per-service auth config, registry-auth forwarding, update options, explicit legacy modes, and the existing rollback path.
  • Six live checks against an isolated Docker 29.9.2 Swarm and a temporary TLS registry: the unmodified script reproduces task replacement when the registry is stopped immediately after a successful manifest preflight; the patched script retains the exact task and image reference under that failure; unchanged references are skipped; a new digest updates successfully; an OCI index retains its top-level digest; and a repeated check of that index does not replace the task. The index is synthetic and only native amd64 execution was tested.

No test framework, CI restructuring, production configuration, or unrelated updater behavior is introduced. I can provide the standalone reproduction steps if useful for review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant