control-plane: data-plane selection at signup + colocated trial bucket (gated)

[SeanWhelan]

Description:

New tenants can now pick a specific public data plane at signup. The betaOnboard directive claims gain an optional requestedDataPlane (full plane catalog name, e.g. ops/dp/public/aws-us-east-1-c1), validated server-side before it becomes the tenant's default data plane in their storage_mappings.

This also lands the backend half of colocated trial buckets (Phase 2 of estuary/sre#29). When the chosen plane is a public AWS plane and COLOCATED_TRIAL_BUCKETS is enabled, provision_tenant derives that plane's colocated S3 bucket instead of gs://estuary-trial, using the same naming formula as est-dry-dock#326 (estuary-trial-{region}-{sha256(plane_name)[:8]}, golden-vector-tested on both sides). The gate stays off in this PR. Flipping it is a separate follow-up once est-dry-dock#326 merges and real public planes have actually converged and gotten a bucket. Until then, and always for GCP/Azure planes, nothing changes: tenants still land on gs://estuary-trial.

Companion PR: estuary/ui#2045, which adds the picker itself. That one must merge and deploy after this one, not before (see notes below).

Workflow steps:

The onboarding UI shows a data-plane picker between the org name field and the survey (companion PR). Submitting without touching it still records an explicit default: aws-us-east-1-c1, or the first available plane if that one isn't present.

The submitted plane is untrusted client input. provision_tenant selects the planes a new tenant may use — public prefix, not closed — and rejects anything outside that set with ProvisionError::PlaneNotSelectable, which the directive turns into invalidClaims naming the rejected plane. If no claim is submitted at all, behavior is identical to today: aws-us-east-1-c1 default, gs://estuary-trial.

To turn on colocated buckets later, set COLOCATED_TRIAL_BUCKETS=1 (or true) on the agent's environment. No code change needed.

Documentation links affected:

None. This is control-plane/agent-internal behavior and no user-facing docs cover the signup bucket-selection mechanics today.

Notes for reviewers:

Deploy order matters here. The Claims struct uses deny_unknown_fields, so this needs to merge and deploy before the UI PR ships. If an old agent gets requestedDataPlane from a new UI, signup fails with invalidClaims naming the unknown field. I checked this failure mode live, not just in theory.

data_planes.closed is the single source of truth for what a new tenant may be placed on, and the migration in this PR marks gcp-us-central1-c1/c2 closed. I started out with a hardcoded exclusion list for those two, which left "selectable" defined in three places that could disagree — the list, the SQL building the tenant's data_planes, and a separate existence query in the agent — while provision_tenant ignored closed altogether. That combination meant closing a plane would hide it from the picker (publicDataPlanes already honors closed) but keep writing it into every new tenant's storage mapping. Now there's one query and one predicate, and retiring a plane is a data change rather than a code change plus a deploy. Existing tenants on those planes are unaffected; closed only governs new selection.

In crates/control-plane-api/src/directives/beta_onboard.rs, provision_tenant now computes plane ordering (order_public_planes) and storage specs (storage_specs) in Rust instead of a single hardcoded CTE. This is behavior-preserving for existing callers (requested_data_plane: None, colocate_trial_bucket: false), confirmed by the integration test's unchanged AcmeTenant snapshot.

trial_bucket_name parses the plane name itself and returns Option<(bucket, region)> rather than taking a region alongside the name, so a bucket can't be derived from a region that disagrees with its plane. parse_data_plane_name and DataPlaneCloudProvider also moved from server::public::graphql::data_planes to crate::data_plane — that logic was living in a GraphQL-presentation module despite being plain domain parsing the directive code needed too.

The golden vector is pinned as a unit test: ops/dp/public/aws-us-east-1-c1 gives estuary-trial-us-east-1-ccc98e22. If this ever fails, the Rust and Python (est-dry-dock) derivations have drifted apart. Fix the drift, not the test.

262 tests across agent and control-plane-api pass, along with cargo sqlx prepare --workspace --check. Note these have to run under cargo nextest — .config/nextest.toml serializes the agent's DB tests via the serial-db-tests group, and under plain cargo test they run in parallel against one database, truncate each other's fixtures, and throw dozens of failures that look like real breakage but aren't.

Worth a look during review: the beta_onboard snapshot gains exactly one case in this PR — a closed public plane rejected with invalidClaims — and every pre-existing case is byte-identical. That diff is the evidence the refactor didn't change behavior.

I also live-tested this end to end on a local stack: a real signup through the actual onboarding form, gate on, plane matching the AWS naming convention, produced a storage_mappings row with the exact derived S3 bucket name. After applying the migration, publicDataPlanes drops the closed plane and the picker renders the rest.

Known follow-ups, deliberately not in this PR: storage_specs still builds its JSON by hand rather than through models::StorageDef, and COLOCATED_TRIAL_BUCKETS is read via an env var rather than a clap Args field. The second is why the colocated S3 path has unit coverage but no end-to-end coverage — worth closing before the gate is flipped on, since the harness could then exercise it.