runtime: forward the config-encryption endpoint to connectors - #3291
Open
mdibaiee wants to merge 1 commit into
Open
runtime: forward the config-encryption endpoint to connectors#3291mdibaiee wants to merge 1 commit into
mdibaiee wants to merge 1 commit into
Conversation
Connectors which emit configUpdate events encrypt the restated config through the config-encryption service, defaulting to Estuary's public one when ENCRYPTION_URL is unset. Nothing set it: connector containers receive only LOG_FORMAT and LOG_LEVEL, so on a local stack a connector reached production and produced a config sealed with keys that stack cannot decrypt. Forward CONFIG_ENCRYPTION_URL from the runtime's own environment as ENCRYPTION_URL, alongside the other runtime knobs already read there, and have the local stack emit it. Connector containers also get --add-host=host.docker.internal:host-gateway for local planes, since that name is how the stack addresses host services and Linux docker does not provide it. Verified on a local stack: the connector container receives ENCRYPTION_URL=http://host.docker.internal:10021/v1/encrypt-config, and a connector built with the production default emitted a configUpdate whose secrets are AGE-encrypted -- the local service's keychain, not production GCP KMS.
mdibaiee
commented
Aug 1, 2026
Comment on lines
+167
to
+170
| // Resolve `host.docker.internal` to the host, so that URLs | ||
| // pointing at host services (e.g. CONFIG_ENCRYPTION_URL) work | ||
| // under Linux docker as well as Docker Desktop. | ||
| "--add-host=host.docker.internal:host-gateway".to_string(), |
Member
Author
There was a problem hiding this comment.
I don't think we want this merged, I'm leaving this PR as a reference for agents to know how to run connectors locally which need config encryption endpoint
This was referenced Aug 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
Connectors that emit
configUpdateevents encrypt the restated config through the config-encryption service, readingENCRYPTION_URLand otherwise falling back to Estuary's public endpoint (go/common/config_update.goin the connectors repo, andestuary-cdk'sbase_capture_connector.py).Nothing ever set that variable.
crates/runtime/src/container.rspasses connector containers onlyLOG_FORMATandLOG_LEVEL, andmise/tasks/local/supabasesetsCONFIG_ENCRYPTION_URLfor the agent, not for connectors. So on a local stack a connector emitting aconfigUpdatecalled production config-encryption and produced a config sealed with keys that stack cannot decrypt.This forwards
CONFIG_ENCRYPTION_URLfrom the runtime's own environment into the container asENCRYPTION_URL, next to the other runtime knobs already read there (CONNECTOR_CGROUP_PARENT,CONNECTOR_MEMORY_LIMIT,DOCKER_CLI), and has the local stack emit it viaemit_common_varsso both the reactor's embedded runtime and the agent have it to pass on. When the variable is unset — production today — behavior is unchanged.Connector containers on local planes also get
--add-host=host.docker.internal:host-gateway. That hostname is how the local stack already addresses host services, but Linux docker doesn't provide it (only Docker Desktop does), and connector containers sit on the flow bridge network where the gateway address isn't fixed.Workflow steps:
No workflow change. After
mise run local:stack, connectors emittingconfigUpdateevents reach the stack's own config-encryption service instead of production.Documentation links affected:
None.
Notes for reviewers:
Verified end-to-end on a local stack, with the connector built carrying the production default URL so that reaching the local service could only come from the forwarded variable:
docker inspecton the running connector container showsENCRYPTION_URL=http://host.docker.internal:10021/v1/encrypt-configandExtraHosts: [host.docker.internal:host-gateway].configUpdateevent's config hascredentials.password_sops = ENC[AES256_GCM,…]with anagesops stanza — the local service's keychain. Production seals with GCP KMS, so this is positive evidence the local endpoint served the request.One step I could not verify in my stack: whether the control plane then applies the update.
config_updatesis populated by the ops catalog materialization, and this stack's ops pipeline is degraded for unrelated reasons (plane-linkand the ops rollup derivations fail becausederive-typescriptis published for amd64 only, so they cannot run on an arm64 VM). The connector-side half — which is what this PR changes — is verified.Depends on nothing, but note #3290 fixes a separate blocker that must be in place for connectors to run at all on an arm64 dev VM.