Skip to content

build: cross-compile flow-connector-init for linux/amd64, bump CONNECTOR_INIT_IMAGE - #3290

Open
mdibaiee wants to merge 1 commit into
masterfrom
mahdi/connector-init-amd64
Open

build: cross-compile flow-connector-init for linux/amd64, bump CONNECTOR_INIT_IMAGE#3290
mdibaiee wants to merge 1 commit into
masterfrom
mahdi/connector-init-amd64

Conversation

@mdibaiee

@mdibaiee mdibaiee commented Aug 1, 2026

Copy link
Copy Markdown
Member

Description:

Two fixes so that a locally built stack can actually run connectors, and so that connectors see the spec fields the runtime sends them.

build:connector-init now builds for linux/amd64. flow-connector-init is bind-mounted into every connector container as its entrypoint, and connector containers always run linux/amd64 (hard-coded in crates/runtime/src/container.rs). The task built for $(uname -m), so on an aarch64 host it produced an aarch64 binary those containers cannot exec — every connector container dies before starting. The only symptom is:

publication failed with status: JobStatus { type: BuildFailed }
  detail: "failed to inspect a started docker container (did it crash?)
      docker command ["inspect", …, "fc_8535aae"] failed: error: no such object: fc_8535aae"

with docker ps -a --filter name=fc_ showing the container stuck in Created. On Apple Silicon this makes a locally built stack unable to run any connector at all.

The target is now x86_64-unknown-linux-musl unconditionally, cross-compiling when the host differs. ring and aws-lc-sys compile C, and those objects must be compiled against musl headers: a glibc cross-gcc (x86_64-linux-gnu-gcc) leaves glibc-only symbols (__isoc23_sscanf, __fprintf_chk) that the musl link cannot resolve. So zig cc is used as the cross C compiler — it ships headers and libc for every target — while rustc's bundled lld links against the target's musl. Two zig quirks are handled by generated wrappers: cc-rs passes an LLVM triple via --target= that zig rejects, and zig cc enables UBSan by default without linking its runtime.

locate_bin() searches PATH, so mise/tasks/local/lib.sh and bootstrap/ide-settings now include the x86_64 musl directory. Without that the runtime silently falls back to CONNECTOR_INIT_IMAGE.

CONNECTOR_INIT_IMAGE bumped from reactor:v0.6.10-62-g8b6aeec1cd3 to reactor:v0.6.12-37-g4cb160dc000 (the newest published tag). The old pin was 67 commits older than 4d66c8d, which added created_at to task specs. Because that binary decodes and re-encodes every request between runtime and connector, a copy older than a protocol field drops it silently — connectors observed specs with no created_at at Validate, Apply and Open even though the build database and live_specs both carried it. The comment now says the pin must be bumped whenever a spec/protocol field is added.

Workflow steps:

mise run build:connector-init now produces an x86_64 binary on any host (~40s cross-compiling on an 8-CPU aarch64 VM). No workflow change beyond that; mise install picks up zig.

Documentation links affected:

None.

Notes for reviewers:

  • Verified on an aarch64 Lima VM: the task produces ELF 64-bit x86-64, the runtime resolves it from PATH, and the container mounts it (md5 of the mounted /flow-connector-init matches the built binary). Connectors then receive the full spec — a materialization reported createdAt="2026-08-01" at Validate, Apply and Open, where before all three saw an absent field.
  • zig is added to mise.toml purely as a cross C compiler; on x86_64 hosts (CI) the build path is unchanged and zig is not invoked.
  • gcc-x86-64-linux-gnu / g++-x86-64-linux-gnu are added to apt-packages-ci-extra. They are not used by the zig path and can be dropped if you prefer — they were the first approach and are harmless to keep for other cross needs.
  • I deliberately did not publish a floating reactor:stable tag; CI tags images only with git describe, and :stable exists solely for derive-python/derive-typescript. Note the newest master commits often have no image at all (docs-only commits skip the platform build), so the newest usable tag lagged master by two commits when I picked it.

flow-connector-init is bind-mounted into every connector container as its
entrypoint, and connector containers always run linux/amd64. The build task
targeted $(uname -m), so on an aarch64 host it produced an aarch64 binary that
those containers cannot exec: every connector dies before starting, reported only
as "failed to inspect a started docker container (did it crash?)" with the
container stuck in Created.

Build for x86_64-unknown-linux-musl unconditionally, cross-compiling when the
host is something else. ring and aws-lc-sys compile C, and their objects must see
musl headers -- a glibc cross-gcc leaves glibc-only symbols (__isoc23_sscanf,
__fprintf_chk) that the musl link cannot resolve -- so use zig cc, which ships
headers and libc for every target, and let rustc's bundled lld link against the
target's musl.

locate_bin() searches PATH, so the local stack and IDE launch configs also need
the x86_64 musl directory on it; without it the runtime silently falls back to
extracting flow-connector-init from CONNECTOR_INIT_IMAGE.

Also bump that pinned image, which was 67 commits older than the commit adding
created_at to task specs. Because it decodes and re-encodes every request between
runtime and connector, a copy older than a protocol field drops it silently:
connectors observed specs with no created_at at Validate, Apply and Open even
though the build database and live_specs both carried it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant