Repository navigation
Conversation
|
Docs preview: https://connectors-pr-5183-estuary-docs.mike-39e.workers.dev Changed pages:
Rebuilt on every push, and the link stays the same. |
|
This is the first working doc preview on the new repo @aeluce |
aeluce
left a comment
There was a problem hiding this comment.
Nice! 🎉
On the content, though: the new "multiple materializations to the same database" section isn't specific to MotherDuck, is it? We could pull that out into more general materialization docs or its own separate guide rather than a connector reference page.
d5117c7 to
a8dc5da
Compare
|
The table is generic: 7 connectors create it, 5 let you move it. Every fence query filters on The generic half moved to estuary/docs#59. Merge that first, this links to it. Two side effects: the "metadata tables" plural is stale everywhere ( I left per-connector behavior out of the concept page on purpose. materialize-redshift is mid-conversion to post-commit apply, where the table stops holding a resumable checkpoint. A connector list would be stale the day that lands, which is also why the MotherDuck migration SQL is now scoped to MotherDuck. |
aeluce
left a comment
There was a problem hiding this comment.
Since this touches on connector specs and Go drivers, even if just on the language side, not sure if you wanted to run this past the integrations team as well.
estuary/flow#3382 merged this into site/docs on 2026-08-28, after the docs cutover. That location no longer publishes, so the content never reached docs.estuary.dev and would have been destroyed when site/ is deleted. Content is unchanged from the flow copy: name flow_checkpoints_v1 as the single metadata table this connector creates, and add the section on running two materializations against one database, where sharing an endpoint database and schema means sharing the checkpoints table. Found by the pre-deletion drift sweep, which reports this as the only flow-ahead file remaining.
Six connector config schemas and ten reference pages said the endpoint schema holds "associated materialization metadata tables". The plural is a leftover from flow_materializations_v2, removed in c9d0ebf; only flow_checkpoints_v1 remains. These strings render in the UI endpoint config form, so the snapshots are regenerated. BigQuery and Snowflake create no metadata table at all (MetaCheckpoints is nil in bigquery.go, and the field is omitted in snowflake.go), so their clause is deleted rather than corrected. The MotherDuck section now says rows are keyed by materialization name and never overwrite each other, and attributes the retry behavior to MotherDuck aborting conflicting transactions rather than queuing them. Its migration SQL is scoped to MotherDuck, which is still destination authoritative: its Acknowledge handler is a no-op. Copying rows does not carry progress on connectors that use post-commit apply.
ad8fc22 to
28dc22b
Compare
Strix Security Review✅ No open security findings. PR summary: Rewords connector configuration descriptions and reference pages to name the single `flow_checkpoints_v1` metadata table instead of stale plural "metadata tables", drops the claim from BigQuery and Snowflake which create none, and replaces the MotherDuck note with a MotherDuck-specific section covering shared checkpoints tables and the procedure for moving a materialization to a different schema. Review summaryThe change is limited to documentation prose, Ruled out (2)
Review 1 on this PR · Reviewed by Strix |
Started as a port of the MotherDuck doc change from estuary/flow#3382, which merged into
site/docsafter the docs cutover and so never reached docs.estuary.dev. Review raised that the new section was not MotherDuck-specific, which turned out to be half right, and widened the scope.Depends on estuary/docs#59, which adds the concept section this links to. Merge that first.
Why
The
/schemadescription on six connectors says the endpoint schema holds "associated materialization metadata tables". The plural is a leftover:flow_materializations_v2was removed in c9d0ebf ("materializations: don't persist a spec in the destination system"), andMetaSpecsno longer exists on theEndpointstruct. Onlyflow_checkpoints_v1remains.For BigQuery and Snowflake the sentence is wrong outright. Neither creates any metadata table:
MetaCheckpointsis nil inmaterialize-bigquery/bigquery.go, and the field is omitted entirely inmaterialize-snowflake/snowflake.go. Both recover from an in-memory runtime checkpoint. Their clause is deleted rather than corrected.These strings are the jsonschema descriptions rendered in the UI endpoint config form, not just docs, which is why the
TestSpecificationsnapshots move with them.What
flow_checkpoints_v1. BigQuery's clause removed..snapshots/TestSpecificationfiles regenerated.materialization=, so rows never collide logically. The failure mode is MotherDuck-specific: DuckDB aborts conflicting transactions rather than blocking, which is whydriver.goretriesduckdb.ErrorTypeTransaction.The migration SQL is kept but scoped to MotherDuck, which is still Destination Authoritative (its
Acknowledgehandler is a no-op). On connectors converting to post-commit apply the table records applied-transaction tokens rather than a resumable checkpoint, so copying rows would not carry progress. It also now says to pause the materialization first, which the original omitted.Verified
All six pass after regenerating snapshots.
go buildclean on the same packages. No occurrence of "materialization metadata tables" remains outside asqlcapturecomment about filtering these tables out of discovery.