feat(import): domain reconciliation + ontos* round-trip on ODCS/ODPS import (#851)

[larsgeorge-db]

Closes #851.

Stacked on #855 (PR #860) — base is feat/import-batch, not development. Review/merge #860 first; this PR's own diff is the domain-reconciliation layer on top.

Best-effort domain reconciliation on import, a lossless round-trip on export, and a per-upload "Create missing domains" toggle (default OFF). Unified across contracts (ODCS) and products (ODPS) via the shared DomainExportAdapter.

Reconciliation (import)

• Match ID first, then unique name. Priority: ontosDomainId (authoritative) → legacy domainIds/primaryDomainId/domainId → standard domain name (+ legacy additionalDomains, still read for backward compat).
• No match → left unassigned. The previous silent auto-create in contracts' _resolve_domain is removed (this also fixes the contracts/products asymmetry). The raw incoming domain string(s) are always preserved as the ontosOriginalDomain provenance custom property.
• "Create missing domains" toggle (default OFF) auto-creates unresolved names and assigns them. Threaded route → manager → adapter; applied per-upload to every entity in the batch. File uploads take a create_missing_domains form field; the paste /odcs/import takes a query param.

Round-trip (export)

• apply_odcs emits ontosDomainId (all assigned ids, primary first — the authoritative rebind key) as a custom property, and keeps the standard domain = primary name. ontosOriginalDomain rides along as a stored property.
• Drops the legacy top-level domainIds/primaryDomainId and customProperties.additionalDomains (superseded by ontosDomainId); both are still read on import so pre-feat(import): adopt ODCS/ODPS domain element on YAML import and round-trip #851 exports keep working.
• merge_custom_properties now preserves ontosDomainId across the export builders' customProperties rebuild.

Latent bug fixed

build_odcs_from_db had four redundant inner import json statements shadowing the module-level import, leaving json unbound and crashing custom-property deserialization. Newly exercised because ontos* values are list-valued (stored as JSON strings, deserialized on export). Removed the inner imports.

Frontend

• "Create missing domains" toggle in the contract upload dialog (covers file drop and pasted JSON), and a new compact product import-options dialog hosting the same toggle. Both post the flag with the upload.

Tests & verification

• Reworked test_domain_export_adapter to the ontosDomainId/ontosOriginalDomain contract; added create_missing, stale-id-fallthrough, and provenance coverage.
• New manager-level test_import_domain_reconcile: match, no-match-unassigned + provenance, create_missing toggle, and export→re-import UUID rebind — for both contracts and products. (35 domain/import unit tests green; 5 existing product domain-resolution tests still green.)
• Live (isolated 8140/3140): export emits ontosDomainId/ontosOriginalDomain; re-importing an exported file rebinds to the same domain UUID. Playwright: both toggles render/flip; a product upload with the toggle ON auto-created the missing domain.

This pull request and its description were written by Isaac.

[mvkonchits-db]

Requesting changes on one authorization issue. Items marked (verified) I confirmed in the code.

Must fix:

1. (verified) domain_export_adapter._create_domain calls domain_repo.create(db=db, obj_in=DataDomainCreate(name=name)) directly. The create_missing_domains toggle therefore lets any user with contract or product write permission create top-level domains, with no data-domains permission check, and created_by is accepted but not passed to the repository, so attribution is lost. Please require data-domains write permission for the toggle (or check it in the route) and record the creator.

Should fix (reported, not traced line by line by me):
2. A partly stale ontosDomainId list returns only the surviving ids with no name fallback. Importing [deleted-primary, valid-secondary] with domain: Primary drops Primary and promotes the secondary.
3. Exporting an entity with no domain assignments does not remove a stored stale ontosDomainId, so re-import reassigns the removed domain.
4. Name reconciliation is exact, untrimmed and case-sensitive (" finance " vs Finance is left unassigned, or creates a duplicate when auto-create is on).

Stack notes: relies on #860's per-entity batch rollback, and #863 builds on this export/custom-property pipeline, so these defects carry into ID-adoption imports.

[mvkonchits-db]

Re-reviewed 4610d15. The permission gate is what I asked for, but the same commit breaks product upload.

Blocker (verified):

• upload_data_products in data_product_routes.py still calls manager.create_products_from_files(file_inputs, ...), but this commit deleted the line that builds file_inputs (file_inputs = [(name, raw) for name, raw, _ in capped]). Every product upload now raises NameError and returns the generic 500. I confirmed with an AST check that file_inputs is read at line 1640 and never assigned in the function, its parameters or the module. It is defined at #860's head, so it looks like a merge resolution slip.
• CI cannot catch this here. A PR whose base is not development only runs the title check, so none of the backend tests ran on this head.
• Please restore the line, and add a route level test for product upload and a test that asserts a 403 when create_missing_domains is on without data-domains write.

Verified as fixed:

• The toggle now requires data-domains write at all three import endpoints, with a clear 403 (the existing handler re-raises HTTPException).
• Auto-created domains record the real creator.
• Name matching is trimmed and case-insensitive.

Notes:

• user_has_feature_level resolves group based permissions only (no team role override). It fails closed, so it is not blocking, but a short comment would help.
• The deferred items (stale ontosDomainId list, clearing a stale id on export) are fine as a follow-up. Please file it.
• #863 and #930 merged an older head of this branch, so they will need a rebase after the fix.

[larsgeorge-db]

Thanks for the catch — all four items addressed in 0266576.

• Restored file_inputs = [(name, raw) for name, raw, _ in capped] right after read_uploads_capped() in upload_data_products. The parser works on raw bytes so no per-file decode step is needed (unlike contracts).
• Added two route-level tests in test_data_product_routes.py: a golden-path minimal-ODPS upload (guards the NameError regression) and a 403 for create_missing_domains=True when the user lacks data-domains write. Both pass locally.
• Expanded user_has_feature_level's docstring to call out the two caveats you noted — group-based resolution only (no team/applied-role override) and fail-closed on error.
• Filed the deferred items as Follow-up: reconcile stale ontosDomainId on re-import + clear on export when domain is gone #944 (stale ontosDomainId on re-import + clearing a stale id on export).

Rebasing #863 and #930 onto this head next so they carry the fix.

[mvkonchits-db]

Re-reviewed 0266576e2. The file_inputs regression is fixed (I re-ran the AST check: it is read at line 1643 and assigned in the function). The commit also adds a real route test for product upload and a 403 test for the data-domains gate, and the user_has_feature_level docstring now states the group-only and fail-closed behavior. Together with the gate, creator attribution and case-insensitive matching from 4610d15c6, everything I asked for on this PR is done. Approving the content.

It cannot merge yet. I retargeted this PR from feat/import-batch to development (that branch is stale now that #860 is merged), and GitHub reports it as conflicting with 25 files. #860 was squash-merged, so the copies of its commits in this branch no longer line up with development. Please rebase onto development and drop the #860 commits, for example git rebase --onto origin/development dccf7c50d feat/import-domain-recon, then push with --force-with-lease. Resolve any overlap with #860's later fixes in favor of what is on development.

Two reasons this matters beyond the conflict:

• The backend tests have never run on this branch, because stacked PRs only run the title check. The file_inputs bug got through for that reason. Once it targets development the full suite runs.
• A push dismisses my approval, so I will re-review the rebased diff and approve again. It should be just the #862 change.

The follow-up for the stale ontosDomainId items is tracked in #944, as noted in the commit. Thanks for the quick turnaround.

[mvkonchits-db]

Re-reviewed the merged result at 44fb41fe6. Approving.

What I checked:

• The diff against development is now 13 files and is only this change. CI ran the full suite on the development base and is green.
• I compared every line the PR adds against the four commits I reviewed earlier (ba05e7bfb, 1f3fb48a7, 4610d15c6, 0266576e2). Nothing in the final diff is outside them, so the conflict resolution added no unreviewed content. The only lines from those commits that are not in the final diff were superseded (the old domain_repo.create call, replaced by the created_by version) or already on development through #860 (the file_inputs line).
• All of my fixes are still present: the data-domains write gate at the three import endpoints, creator attribution, trimmed and case-insensitive matching, and the restored file_inputs assignment (read at line 1643, assigned in the function).
• pyflakes finds no new undefined names. The two it reports (DataContractRead, DataProductDb) are string forward references that already exist on development.
• The import json lines removed from build_odcs_from_db are fine, since json is imported at module level.

One behavior change to be aware of, intended and safer: the old silent auto-create of a missing domain by name (as system) in _resolve_domain is gone. An unknown domain name now leaves the entity unassigned unless the explicit create-missing toggle is on.

#863 builds on this branch, so it will need to move onto development once this lands.