[aljoscha]

Motivation

DDL transactions currently reject any intervening catalog revision.
A transaction renaming one view can therefore fail because another
session creates an unrelated object.

Simply removing that check is unsafe. Precomputed writes can refer to
objects that have disappeared, miss newly created dependents, or rely
on revoked permissions. Commit must also preserve catalog results
already exposed to the client.

Description

Retain supported ALTER ... RENAME and ALTER ... SWAP statements with
their parameters and per-statement session context. Use the same command
logic for speculative execution and atomic application at commit.

At commit, evaluate the statements in order against the current catalog,
including preceding statements in the batch. Recompute name resolution,
authorization, and dependent-object rewrites there. Missing targets,
destination collisions, and authorization failures reject the entire
batch without publishing partial changes.

Names are deliberately resolved again at commit. If another session
drops a target and creates a replacement with the same name, the command
can act on that replacement, provided the outcomes already reported to
the client remain unchanged.

Preserve each rename's reported object type, change/no-op outcome, and
planning notices. An ALTER ... IF EXISTS that reported a missing object
cannot silently become a successful rename at commit.

Retain supported SHOW CREATE rows and meaningful statement descriptions
from the start of the transaction. If the transaction enters DDL mode,
validate those earlier inspections before its first mutation. Recompute
retained results at their original statement positions during commit,
not against the final catalog.
Successfully recomputed results that differ produce a serialization failure.
Replanning errors, such as a missing object, surface as they would in serial
execution. Equivalent results
remain acceptable even if object identity changes.

EXPLAIN and supported transactional DDL cannot be mixed in either order.
EXPLAIN results are not replayed, so executing EXPLAIN prevents that
transaction from subsequently entering DDL mode.

Inspections are retained only while a DDL commit remains possible.
Entering a read, write, subscription, or single-statement mode, or
executing EXPLAIN, releases buffered inspections and stops retaining
new ones.

Scope

Remove the blanket catalog-revision check only for batches whose
mutating statements are supported renames and swaps. Other mutating DDL
retains that check, including source creation that reports an IF NOT EXISTS
no-op before or after a rename. No additional statement families become eligible
for multi-statement transactions.

SHOW CREATE and statement descriptions do not themselves enter DDL
mode or restrict subsequent statements. Transactions that never enter
DDL mode do not validate retained inspections. Ordinary data reads,
object listings, and other SHOW forms remain outside this protection.
Serving a cached portal description without recomputation does not
add a retained check.

Coverage

The rename-focused CI workflow covers:

• Unrelated concurrent DDL and newly created dependents.
• Replacement and missing targets, destination collisions, revoked
  permissions, and complete rollback, compared with serial execution.
• Reported no-op outcomes, schema swaps, and concurrent rename batches.
• Source and source-fed-table no-ops followed by concurrent schema renames
  or dropped collision targets, with unchanged published state on failure.
• Catalog results and statement descriptions, including changed
  definitions, equivalent replacements, statement-position semantics,
  protocol preparation, cursor wrappers, inspection before DDL, and
  unchanged non-DDL transaction behavior.
• EXPLAIN/DDL rejection in both orders through direct, prepared, and
  implicit execution, with transaction-state cleanup.

A session lifetime test checks that inspections are released when DDL
becomes impossible and that later inspections are not retained.

SQL tests also cover per-statement search paths and independent
concurrent renames.

[def-]

QA LLM Review

1. HIGH -- Rename after an IF NOT EXISTS no-op dry-runs against a stale state and panics the coordinator

src/adapter/src/coord/ddl.rs:384

When a transaction's first DDL statement is a CREATE TABLE ... FROM SOURCE or CREATE SOURCE with IF NOT EXISTS that hits an existing object, nothing is recorded: no op and no dry-run snapshot. A later rename then counts as a rename-only batch and skips the revision check, so it is dry-run against the stale start-of-transaction state but a fresh durable transaction. If another session renamed the same schema, cluster or replica in the meantime, apply_inverted_lookup asserts and environmentd aborts. One user with two connections can trigger this.

Details

Reproduced at e0a3ea1 with a debug sqllogictest build. On the base commit 03f057c the same script fails cleanly with DDLTransactionRace.

CREATE SOURCE lg FROM LOAD GENERATOR COUNTER;
CREATE TABLE lg_t FROM SOURCE lg;
CREATE SCHEMA sc;
BEGIN;
CREATE TABLE IF NOT EXISTS lg_t FROM SOURCE lg;  -- no-op, snapshot stays None
-- other connection: ALTER SCHEMA sc RENAME TO sc_b;
ALTER SCHEMA sc RENAME TO sc_a;
-- thread 'coordinator' panicked at src/adapter/src/catalog/apply.rs:2617:13:
-- retraction does not match existing value: "sc_b"

Mechanism: the DDL-mode entry at src/adapter/src/coord/command_handler.rs:1426 stores state with snapshot: None. The IF NOT EXISTS path (sequence_create_table / sequence_create_source) turns the dry run's ItemAlreadyExists into a notice before add_ops runs. Before this PR, the next statement's add_ops(DDL { revision: current }) raised DDLTransactionRace. Now the !is_ddl() guard skips that call, and the "all commands" predicate is true because the no-op left nothing in ops. transact_incremental_dry_run then opens storage.transaction() at the current durable revision but applies its diffs to the old txn_state. Renames of items survive this mismatch. Renames of schemas, clusters and cluster replicas look up the old durable name and panic.

The same missing trace also skips validation of the reported outcome, which contradicts the PR's statement that other mutating DDL keeps the revision check. I verified BEGIN; CREATE TABLE IF NOT EXISTS lg_t FROM SOURCE lg; ALTER VIEW v RENAME TO w; COMMIT with a concurrent DROP TABLE lg_t: it commits even though the client was told lg_t already exists.

Suggested fix: mark the transaction when DDL mode is entered by any statement other than a rename or swap, for example with a flag on TransactionOps::DDL set in handle_execute_inner. Refuse the bypass in catalog_transact_with_ddl_transaction and sequence_end_transaction_inner when that flag is set. As a further guard, treat txn_snapshot == None with a changed revision as DDLTransactionRace, or rebase txn_state on the live catalog, instead of dry-running a stale state against fresh durable data.

[ggevay]

Releasing some Claude comments for now.

---

The QA bot's finding reproduces on the CI image of e0a3ea1; on #40007's head the same sequences fail cleanly with 40001. Two more triggers of the same panic:

• CREATE SOURCE IF NOT EXISTS on an existing source as the no-op.
• ALTER SCHEMA a SWAP WITH b as the later statement, while another session renames a.

The skipped outcome also isn't checked when the no-op comes later: BEGIN; ALTER VIEW v RENAME TO w; CREATE TABLE IF NOT EXISTS t FROM SOURCE s;, then another session drops t, and COMMIT succeeds.

On the later layers' CI images, the panic is gone from #40013 on (the snapshot is captured when the transaction enters DDL mode), and the unchecked skip from #40015 on (it fails with 40001). So landing #40008 before #40013 would ship the crash.

Minor, on the description: "Changed results produce a serialization failure" could also say that a retained result whose object was dropped fails with the error serial execution would give, for example unknown catalog item 'x'.

[ggevay]

Nit: from "Pending external work" on, this describes external preparation and connection handling, which only arrive in #40015-#40017. Since this PR can merge on its own, maybe move those sentences to the PRs that add that code?

[ggevay]

A few smaller things on top of the earlier comments.

[ggevay]

This error is also raised when a retained SHOW CREATE row or statement description no longer matches (CatalogResultCommand::execute, DescriptionCommand::execute), so "a DDL statement" can point the user at the wrong statement. Maybe:

Suggested change

	f.write_str("a DDL statement would produce a different outcome at commit")
	f.write_str("a statement in this transaction would produce a different result at commit")

[ggevay]

Nit: the same six EXPLAIN variants are listed again in frontend_peek.rs, before the other record_explain() call. An EXPLAIN variant added to only one list would, on the other path, neither be rejected in a DDL transaction nor block later DDL. Maybe one shared is_explain(&Statement) next to record_explain?

[ggevay]

describe now also retains the description in the session, which is why it needs &mut Session. A knock-on: this function and verify_prepared_statement now clone the statement and its description on every call, before the revision check, where they used to borrow them. Could they compare revisions first and clone only when re-describing?

[aljoscha]

Aljoscha the human here: thanks, I'll fix these, and sorry, some of these came up from chopping the change up into PRs, as you can see from how the bug is fixed in later PRs. 😅 But I should and will totally fix the issue locally here in that PR!

[ggevay]

Thanks, LGTM!