deploy: ontokit-deploy requires AUTH_MODE in .env, but the real source of truth is compose.yaml — every deploy refuses

[damienriehl]

Symptom

/usr/local/sbin/ontokit-deploy deploy <api-sha> <web-sha> refuses on a correctly-configured DEV box:

/usr/local/sbin/ontokit-deploy: line 151: AUTH_MODE: AUTH_MODE must be set in /opt/ontokit/.env

Hit on DEV (CPX41, 178.156.208.239) 2026-08-13 attempting to move DEV from api 20cb6aa7/web cfa91623 to the feat/pr-party heads.

Root cause

The hardening merged in chore/deploy-workflow-hardening added:

: "${AUTH_MODE:?AUTH_MODE must be set in $ENV_FILE}"

But AUTH_MODE was never in .env on the box. The live U7 flip set it in compose.yaml, where it appears three times as AUTH_MODE: optional (service env + build args). The running containers confirm it: docker exec ontokit-api-1 printenv AUTH_MODE → optional.

So the script demands the key from a file that never held it, while the value it wants sits in the file it is about to invoke. Any box configured the way the runbook actually configured it cannot deploy.

Second, worse bug: the failure leaves a false-green state

deploy_pair checks out the new SHAs before sourcing .env and asserting the required keys. When the assertion fails:

• both repos are already git checkout --detach'd to the new SHAs;
• no image is rebuilt, so containers keep running the old build;
• status reads git HEAD, not the running images, so it cheerfully reports the new SHAs as deployed.

Observed exactly this: status returned api=6bec76ae… web=59747361… while docker ps still showed both containers "Up 3 days". A monitor trusting status would call the deploy successful.

Suggested fixes

1. Validate before mutating. Source .env and assert every required key before the first git checkout --detach, so a refusal is a no-op.
2. Take AUTH_MODE from the right place. Either read the effective value from docker compose config, or default to compose's value and treat .env as an override — rather than requiring duplication of a value compose already owns.
3. Make status tell the truth. Report the SHA baked into the running images (or compare git HEAD against them and flag drift), not just git rev-parse HEAD.
4. Document it in deploy/RUNBOOK.md and deploy/.env.example if .env really is meant to be authoritative.

Workaround applied on DEV

Appended AUTH_MODE=optional to /opt/ontokit/.env (matching compose and the live state exactly; backup at .env.bak-preauthmode-20260813) and re-ran the deploy.

Related

U7 flip and DEV runbook: docs/roundup-2026-08/DEV-RUNBOOK.md. Companion to the "silence is not success" and "a self-check that can pass via another credential proves nothing" learnings — same family: a verification surface that reports success it did not establish.