feat(import): adopt YAML entity UUID as primary key + duplicate handling (#853)

[larsgeorge-db]

Closes #853.

Stacked on #851 (PR #862) → #855 (PR #860). Base is feat/import-domain-recon. Merge #860 then #862 first; this PR's own diff is the id-adoption layer.

Adopt a valid, non-colliding UUID from the imported ODCS/ODPS file as the entity's primary key so YAML-expressed cross-entity links (product↔contract) survive import, with a lossless id round-trip on export.

Adoption (both entities)

• create_product / create_from_upload take adopt_ids (default on): a valid UUID not already present in that entity's table is adopted as the PK; a non-UUID or a colliding/adopt-off id falls back to a generated UUID (today's behavior).
• The incoming id is always recorded as the ontosOriginalId provenance custom property; the legacy sourceId is still written for non-UUID ids during the transition (read either). Export re-derives ontosEntityId (current PK).

Duplicate-id handling (batch)

• create_*_from_files take on_duplicate = "skip" (default) | "new". A payload id that is a valid UUID already present is either recorded as skipped ("already present (same id)" — making re-import idempotent) or imported as a new copy with a fresh UUID. Import stays create-only. Uses the skipped status already in BatchImportResult (feat(import): support single and batch (multi-entity) uploads for both ODCS and ODPS #855).

Routes / frontend

• Upload endpoints take adopt_ids + on_duplicate form fields; the paste /odcs/import takes them as query params. Both import dialogs gain "Adopt IDs from file" (default on) and "Import duplicate IDs as new copies" (default off) toggles; success toasts report skipped counts.

Bonus bug fix — product ODPS export was crashing

build_odps_export accessed the ODPS API model by validation-alias names (product.api_version, input_ports, output_ports, support_channels, custom_properties, authoritative_definitions, port.contract_id) that aren't attributes on the camelCase model — so GET /data-products/{id}/odps/export raised AttributeError for every product (no test covered it). Corrected to the real field names; the endpoint now works and carries the ontos* keys.

Tests & verification

• New test_import_id_adopt: adopt valid UUID, non-UUID fallback + sourceId/ontosOriginalId, ontosEntityId on export, duplicate skip (default) and import-as-new — for both entities (10 tests). Made the domain-export-adapter test's domain fixture get-or-create so the suite is order-independent (managers commit → rows persist across tests).
• Live (isolated 8140/3140): adoption keeps the file UUID; duplicate skip/new both work; product + contract exports emit ontosOriginalId/ontosEntityId; all three import toggles render with correct defaults.

This pull request and its description were written by Isaac.

[mvkonchits-db]

Requesting changes. Import stays create-only, so an importer cannot overwrite or hijack an existing entity, which is good. Items marked (verified) I confirmed in the code.

Must fix:

1. (verified) ID squatting. adopt_ids defaults to True and any user with write can claim an arbitrary UUID. A legitimate entity imported later with that id is then skipped, and anything resolving references by id (see #864) resolves to the squatter's entity. Please make adoption opt-in, or restrict it to a stronger permission.
2. (verified) UUIDs are validated with uuid.UUID(...) but stored and compared as the raw string (adopted_id = original_id, exact-match repo.get). Uppercase, braced or hyphenless spellings pass validation, miss the duplicate lookup, and create a second identity for the same logical UUID. Canonicalize before lookup and storage.
3. (verified) on_duplicate is a free Form("skip") string. Any value other than exactly "skip" creates a copy, so a typo silently duplicates. Validate against skip / new.
4. (verified by absence) on_duplicate="new" assigns a fresh id but nothing remaps references that still contain the supplied id, so imported ports keep pointing at the pre-existing entity.

Should fix:
5. (verified) A skipped duplicate returns the existing row's id (entity_id=source_id, "already present (same id)") with no visibility check, which acts as an existence oracle.
6. Raw exception text in per-item messages (see the note on #860).

[mvkonchits-db]

Re-reviewed the changes since my last review (a086bc68f, ae844340f, 3d7ebce61, plus the original fab391b21). All of my items are addressed in the code:

• ID squatting: adopt_ids now defaults to false on all three import endpoints. Adoption also needs the new can_adopt_entity_ids role privilege or ADMIN on the feature, and non-privileged callers fall back to a generated id with ontosOriginalId kept. The audit entry records both the requested and the effective value.
• UUIDs are canonicalized before the duplicate lookup and before storage.
• on_duplicate is validated at the route (400 unless skip or new).
• A skipped duplicate no longer returns the existing row's id or name unless the caller is a feature admin.
• Reference remap: I accept your reasoning that nothing needs remapping inside one batch. What remains is the cross-endpoint case (a product imported later whose port points at the old contract id after on_duplicate=new). That is a limitation, not a defect, and your comment documents it.
• The new migration b1_add_adopt_entity_ids is a single head, descends from n1_search_documents, and the revision id is 23 characters. I checked the head chain against current development.

Notes, none blocking:

• user_can_adopt_entity_ids resolves roles by group, so a user granted a role only through assigned_users will not get the privilege. That fails closed, but you may want to include them.
• The unguarded add_column is fine, since fresh databases use create_all plus stamp.
• 3d7ebce61 is a WIP commit; it is harmless under a squash merge.

I am not approving yet, for two reasons. This PR's diff against its current base is 117 files, mostly development content merged into the branch, and I have not inspected those merge resolutions. An unnoticed merge resolution is exactly how file_inputs disappeared on #862. And the backend tests and the Alembic check have not run on this branch, because stacked PRs only run the title check.

Plan: once #862 is rebased and merged, rebase this branch onto development (for example git rebase --onto origin/development <old #862 tip> feat/import-id-adopt), so the diff is just this change and the full CI runs. I will re-review and approve then.

[larsgeorge-db]

Thanks for the detailed re-review. Agreed on both the plan and the two non-blocking notes.

On the retarget:
Will do exactly as you proposed — once #862 merges into development, I'll run git rebase --onto origin/development <old #862 tip> feat/import-id-adopt and retarget this PR to development, so the diff is just this feature (adopt-ids / can_adopt_entity_ids / UUID canonicalization / on_duplicate validation / existence-oracle redaction) and the backend tests + Alembic check actually run on CI. Holding off from touching anything on this branch in the meantime so you don't review stale deltas.

On can_adopt_entity_ids + assigned_users:
Fair point that a user granted a role only through assigned_users currently misses the privilege (same cliff is_user_admin has). I'll fold assigned_users into the lookup in a follow-up — not blocking this PR, but worth doing for parity with how roles are actually assigned in practice.

On 3d7ebce61 (WIP commit):
Confirmed harmless under squash; its diff (the DB column add) is a strict subset of the net-effect feature commit, so a squash merge produces the correct final tree.

On the cross-endpoint remap case:
Noted — a product imported later whose port points at the pre-on_duplicate=new contract id will see a dangling reference. Documented in the manager scaffold-note; could add a one-shot repair later, but that's an "import ordering" artifact rather than a defect in this PR.

[mvkonchits-db]

I rebased feat/import-id-adopt onto development (force-push, 9a55cc44 to 620f4d6d) and retargeted this PR to development, so the full CI now runs against it. The diff is 25 files, which is just this change.

If you have this branch checked out locally, please reset to the remote instead of pushing your old history: git fetch origin && git checkout feat/import-id-adopt && git reset --hard origin/feat/import-id-adopt. The previous tip was 9a55cc44ad3fb269343c03b205e3368ef68e57e7, still reachable by SHA if you need anything from it.

What the branch contains now: your four commits (fab391b2, the WIP column commit, the role privilege, and the gate/canonicalization/validation fix), replayed in order with authorship and messages unchanged, plus one new commit at the front.

The new commit carries the productType marker for _PRODUCT_MARKERS in entity_kind.py. That change only existed inside your merge commit 7789b2913, not in any regular commit, so a plain replay of the commits would have dropped it. It is needed: the #853 tests build products as {name, version, productType} with no kind and no ports, and without the marker #860's cross-type guard skips them as unrecognizable. It is its own commit so it is visible in review. I credited you as co-author. Drop or reword it if you would rather handle it differently.

Conflicts (four hunks, all caused by #860 and #862 having been squash-merged):

• In both managers the wrong-type guard now runs first and the duplicate-id handling follows it, which matches your resolution. I kept your comment noting the ordering.
• In data-products.tsx I kept development's unimportedItems handling, which already covers skipped items.
• In data_product_routes.py I kept the file_inputs line as it is on development.

How I checked it: the rebased tree differs from your previous tip only in files changed by #864 and #939 after your last merge. None of the #853 code (routes, authorization, role privilege, migration, locales, tests) differs. All changed Python files compile, pyflakes reports no new undefined names, and Alembic is a single head (b1_add_adopt_entity_ids on top of n1_search_documents). I did not run the backend tests locally; CI is running now.

I will re-review the diff once CI is green.

[mvkonchits-db]

Re-reviewed the rebased branch at 620f4d6d. Approving.

• The diff against development is 25 files and is only this change.
• CI ran the full suite on the development base: Backend Tests, Frontend Tests, TypeScript, Frontend Build, Alembic Single-Head and Requirements Lockfile all pass. The Coverage Report aggregate job was still running when I checked; it is not one of the required checks.
• I compared every line the PR adds against the commits I reviewed earlier (fab391b2, 3d7ebce6, a086bc68, ae844340). The only added lines outside them are the carried-over productType marker commit and a four-line comment on guard ordering. Lines from the earlier commits that are missing were superseded by the later fix commit or dropped in the conflict resolution.
• All five of my earlier items are in place: adopt_ids defaults to off and needs the can_adopt_entity_ids privilege or feature ADMIN, UUIDs are canonicalized before the duplicate lookup and before storage, on_duplicate is validated, a skipped duplicate only reveals the existing row to feature admins, and the reference-remap question is documented.
• The migration b1_add_adopt_entity_ids is a single head on top of n1_search_documents, and the revision id is 23 characters.

Non-blocking, unchanged from my earlier note: user_can_adopt_entity_ids resolves roles by group only, so a user granted a role only through assigned_users will not get the privilege. That fails closed.