Skip to content

docs: name the checkpoints table, drop the metadata-tables plural - #5183

Open
jwhartley wants to merge 4 commits into
mainfrom
james/motherduck-metadata-table-port
Open

jwhartley wants to merge 4 commits into
mainfrom
james/motherduck-metadata-table-port

Conversation

@jwhartley

@jwhartley jwhartley commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

Started as a port of the MotherDuck doc change from estuary/flow#3382, which merged into site/docs after the docs cutover and so never reached docs.estuary.dev. Review raised that the new section was not MotherDuck-specific, which turned out to be half right, and widened the scope.

Depends on estuary/docs#59, which adds the concept section this links to. Merge that first.

Why

The /schema description on six connectors says the endpoint schema holds "associated materialization metadata tables". The plural is a leftover: flow_materializations_v2 was removed in c9d0ebf ("materializations: don't persist a spec in the destination system"), and MetaSpecs no longer exists on the Endpoint struct. Only flow_checkpoints_v1 remains.

For BigQuery and Snowflake the sentence is wrong outright. Neither creates any metadata table: MetaCheckpoints is nil in materialize-bigquery/bigquery.go, and the field is omitted entirely in materialize-snowflake/snowflake.go. Both recover from an in-memory runtime checkpoint. Their clause is deleted rather than corrected.

These strings are the jsonschema descriptions rendered in the UI endpoint config form, not just docs, which is why the TestSpecification snapshots move with them.

What

  • Six Go config schemas: singular, naming flow_checkpoints_v1. BigQuery's clause removed.
  • Ten reference pages: same fix, linking to the concept section. Snowflake.md and BigQuery.md have the clause removed.
  • Six .snapshots/TestSpecification files regenerated.
  • The MotherDuck section reworded. It previously said sharing the table "can cause write contention and failed transactions", which reads as general Estuary behavior. Every fence query filters on materialization=, so rows never collide logically. The failure mode is MotherDuck-specific: DuckDB aborts conflicting transactions rather than blocking, which is why driver.go retries duckdb.ErrorTypeTransaction.

The migration SQL is kept but scoped to MotherDuck, which is still Destination Authoritative (its Acknowledge handler is a no-op). On connectors converting to post-commit apply the table records applied-transaction tokens rather than a resumable checkpoint, so copying rows would not carry progress. It also now says to pause the materialization first, which the original omitted.

Verified

go test -run TestSpecification ./materialize-postgres/... ./materialize-redshift/... ./materialize-motherduck/... ./materialize-azure-fabric-warehouse/... ./materialize-sqlserver/... ./materialize-bigquery/...

All six pass after regenerating snapshots. go build clean on the same packages. No occurrence of "materialization metadata tables" remains outside a sqlcapture comment about filtering these tables out of discovery.

@estuary-docs-previews

estuary-docs-previews Bot commented Sep 3, 2026 •

Copy link
Copy Markdown

@jwhartley
jwhartley requested a review from aeluce September 3, 2026 04:10
@jwhartley

Copy link
Copy Markdown
Contributor Author

This is the first working doc preview on the new repo @aeluce

@aeluce aeluce left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! 🎉

On the content, though: the new "multiple materializations to the same database" section isn't specific to MotherDuck, is it? We could pull that out into more general materialization docs or its own separate guide rather than a connector reference page.

@jwhartley
jwhartley force-pushed the james/motherduck-metadata-table-port branch from d5117c7 to a8dc5da Compare September 7, 2026 23:14
@jwhartley jwhartley changed the title docs: port the MotherDuck metadata-table section from flow/site docs: name the checkpoints table, drop the metadata-tables plural Sep 7, 2026
@jwhartley

jwhartley commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor Author

The table is generic: 7 connectors create it, 5 let you move it. Every fence query filters on materialization= (materialize-sql/std_sql.go:62-107), so shared rows never collide. MotherDuck is the bad case only because DuckDB aborts conflicting transactions instead of blocking, hence the duckdb.ErrorTypeTransaction retry. Postgres and Redshift would just wait. So I kept a MotherDuck-specific section and cut the general claim.

The generic half moved to estuary/docs#59. Merge that first, this links to it.

Two side effects: the "metadata tables" plural is stale everywhere (flow_materializations_v2 went away in c9d0ebf), and on BigQuery and Snowflake it's wrong outright since neither creates one, so I deleted the clause there.

I left per-connector behavior out of the concept page on purpose. materialize-redshift is mid-conversion to post-commit apply, where the table stops holding a resumable checkpoint. A connector list would be stale the day that lands, which is also why the MotherDuck migration SQL is now scoped to MotherDuck.

@jwhartley
jwhartley requested a review from aeluce September 7, 2026 23:53
aeluce
aeluce previously approved these changes Sep 9, 2026

@aeluce aeluce left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this touches on connector specs and Go drivers, even if just on the language side, not sure if you wanted to run this past the integrations team as well.

estuary/flow#3382 merged this into site/docs on 2026-08-28, after the docs
cutover. That location no longer publishes, so the content never reached
docs.estuary.dev and would have been destroyed when site/ is deleted.

Content is unchanged from the flow copy: name flow_checkpoints_v1 as the single
metadata table this connector creates, and add the section on running two
materializations against one database, where sharing an endpoint database and
schema means sharing the checkpoints table.

Found by the pre-deletion drift sweep, which reports this as the only
flow-ahead file remaining.
Six connector config schemas and ten reference pages said the endpoint
schema holds "associated materialization metadata tables". The plural is
a leftover from flow_materializations_v2, removed in c9d0ebf; only
flow_checkpoints_v1 remains. These strings render in the UI endpoint
config form, so the snapshots are regenerated.

BigQuery and Snowflake create no metadata table at all (MetaCheckpoints
is nil in bigquery.go, and the field is omitted in snowflake.go), so
their clause is deleted rather than corrected.

The MotherDuck section now says rows are keyed by materialization name
and never overwrite each other, and attributes the retry behavior to
MotherDuck aborting conflicting transactions rather than queuing them.
Its migration SQL is scoped to MotherDuck, which is still destination
authoritative: its Acknowledge handler is a no-op. Copying rows does not
carry progress on connectors that use post-commit apply.
@jwhartley
jwhartley force-pushed the james/motherduck-metadata-table-port branch from ad8fc22 to 28dc22b Compare September 30, 2026 02:49
@jwhartley
jwhartley requested a review from aeluce October 2, 2026 01:48
@strix-security

strix-security Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Strix Security Review

✅ No open security findings.

PR summary: Rewords connector configuration descriptions and reference pages to name the single `flow_checkpoints_v1` metadata table instead of stale plural "metadata tables", drops the claim from BigQuery and Snowflake which create none, and replaces the MotherDuck note with a MotherDuck-specific section covering shared checkpoints tables and the procedure for moving a materialization to a different schema.

Review summary

The change is limited to documentation prose, jsonschema description strings on six connector config structs, and the corresponding regenerated TestSpecification golden files. No executable logic, query construction, authentication or input-handling code is touched; the only behavioural surface is the descriptive text rendered in the endpoint configuration form, and each modified struct tag still parses to a single well-formed description field. No security issues were identified in the changed code. The docs site build failure noted in the PR conversation is a broken-link/reference issue against a dependent change and is outside the scope of this security review.

Ruled out (2)
  • Could a changed `jsonschema` description string break or corrupt how the endpoint config specification is parsed, letting malformed fields into the rendered fo…
    Each modified tag was checked and splits into its expected fields with exactly one intact `description=` value, and a Semgrep golang pass over the six files returned nothing.
  • Does the MotherDuck migration snippet copy checkpoint rows between connectors in a way that leaks or corrupts another tenant's progress?
    The added SQL is documentation for an operator to run with their own credentials against their own database, and it is explicitly scoped to MotherDuck with a `materialization =` filter and a note that other connectors store different contents.

Review 1 on this PR · 60c42dc · requested Oct 22 15:02 UTC on a new commit


Reviewed by Strix
Re-run review · Configure security review settings

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants