Repository navigation
feat(import): unify ODCS/ODPS upload dialogs into one shared component - #930
Merged
Merged
Conversation
…ODCS & ODPS (#855) Unify contract (ODCS) and product (ODPS) upload so both accept a single entity, an array of entities in one file, and multiple files in one operation, reported via a truthful BatchImportResult summary (N created / M skipped / K failed). Backend - New shared models.import_results.BatchImportResult / ImportItemResult. The `skipped` status and source_id/entity_id provenance fields are part of the contract from the start so the sibling import features (#851 domain reconciliation, #853 UUID-as-PK duplicate handling) can populate them without another response-shape change. - Contracts: add parse_uploaded_entities (array-in-file parity) and create_contracts_from_files; /data-contracts/upload now takes List[UploadFile], /odcs/import accepts an object or an array. Both return BatchImportResult. - Products: extract _parse_products_content, add create_products_from_files; /data-products/upload takes List[UploadFile] and returns BatchImportResult. Fix a latent bug where json.JSONDecodeError was referenced in an except clause before json was imported (only reachable now that non-json files flow through). - Per-entity failures are recorded as failed items and never abort the batch (each entity commits in its own transaction). Frontend - Shared types/import-results.ts. data-products and data-contracts views send every selected file under `files` (multi-file), and render the truthful summary + per-entity failures. Tests - New manager-level unit tests for both entities (array-in-file, multi-file, bad-entity-continues, unsupported/unparseable file, sourceId preservation). - Updated the contract upload/import integration tests to the new files/BatchImportResult contract. Session: claude -r 37a1420d-86af-4380-bef3-0b74c6f8b4b9
…import (#851) Best-effort domain reconciliation on import, lossless round-trip on export, and a per-upload "Create missing domains" toggle. Stacked on #855 (feat/import-batch). Reconciliation (import, both entities via the shared DomainExportAdapter) - Match ID first, then unique name. Priority: ontosDomainId (authoritative) → legacy domainIds/primaryDomainId/domainId → standard `domain` name (+ legacy additionalDomains, still read for backward compat). - No match → left UNASSIGNED (the previous silent auto-create in contracts' _resolve_domain is removed). The raw incoming domain string(s) are always preserved as the `ontosOriginalDomain` provenance custom property. - Opt-in "Create missing domains" toggle (default OFF) auto-creates unresolved names by name and assigns them. Threaded route → manager → adapter; applied per-upload to every entity in the batch. Contract file upload + product upload take a `create_missing_domains` form field; the paste /odcs/import takes a query param. Round-trip (export, both entities) - apply_odcs now emits `ontosDomainId` (all assigned ids, primary first — the authoritative rebind key) as a custom property, and keeps the standard `domain` = primary name. `ontosOriginalDomain` rides along as a stored property. - Drops the legacy top-level `domainIds`/`primaryDomainId` and the `customProperties.additionalDomains` entry (superseded by `ontosDomainId`); both are still READ on import so pre-#851 exports keep working. - merge_custom_properties now preserves `ontosDomainId` across the export builders' customProperties rebuild. Also fixes a latent `json` UnboundLocalError in build_odcs_from_db: four redundant inner `import json` statements shadowed the module-level import, crashing the custom- properties deserialization — newly exercised because ontos* values are list-valued (stored as JSON strings, deserialized on export). Frontend - "Create missing domains" toggle in the contract upload dialog (covers file drop and pasted JSON) and a new compact product import-options dialog hosting the same toggle; both post the flag with the upload. Tests - Reworked test_domain_export_adapter to the ontosDomainId/ontosOriginalDomain contract; added create_missing + stale-id-fallthrough + provenance coverage. - New manager-level test_import_domain_reconcile: match, no-match-unassigned + provenance, create_missing toggle, and export→re-import UUID rebind, for both contracts and products. Verified live (isolated 8140/3140): API round-trip rebinds by UUID; UI toggle auto-creates a missing domain on product upload. Session: claude -r 37a1420d-86af-4380-bef3-0b74c6f8b4b9
…ing (#853) Adopt a valid, non-colliding UUID from the imported ODCS/ODPS file as the entity's primary key so YAML-expressed cross-entity links (product↔contract) survive import, with a lossless id round-trip on export. Stacked on #851 (feat/import-domain-recon). Adoption (both entities) - create_product / create_from_upload take adopt_ids (default on): a valid UUID that does not already exist in that entity's table is adopted as the PK; a non-UUID or a colliding/adopt-off id falls back to a generated UUID (today's behavior). Contracts adopt via id= on DataContractDb; products via product_data['id']. - The incoming id is always recorded as the ontosOriginalId provenance custom property; the legacy sourceId is still written for non-UUID ids during the transition (read either). Export re-derives ontosEntityId (current PK) as a custom property. Duplicate-id handling (batch, both entities) - create_*_from_files take on_duplicate = "skip" (default) | "new". A payload id that is a valid UUID already present is either recorded as skipped ("already present (same id)") — making re-import idempotent — or imported as a new copy with a fresh UUID. Import stays create-only. Uses the skipped status already in BatchImportResult (#855). Routes/frontend - upload endpoints take adopt_ids + on_duplicate form fields; the paste /odcs/import takes them as query params. Both import dialogs gain "Adopt IDs from file" (default on) and "Import duplicate IDs as new copies" (default off) toggles; success toasts now report skipped counts. Also fixes build_odps_export: it accessed the ODPS API model by validation-alias names (product.api_version, input_ports, output_ports, support_channels, custom_properties, authoritative_definitions, port.contract_id) that are not attributes on the camelCase model, so the product ODPS export endpoint crashed with AttributeError for every product. Corrected to the real field names; the endpoint now works (and carries the ontos* keys). Tests - New test_import_id_adopt: adopt valid UUID, non-UUID fallback + sourceId/ontosOriginalId, ontosEntityId on export, duplicate skip (default) and import-as-new — for both entities. - Made the domain-export-adapter test's domain fixture get-or-create so the suite is order-independent (managers commit, so rows persist across tests in the shared session). Verified live (isolated 8140/3140): adoption keeps the file UUID; duplicate skip/new both work; product + contract exports emit ontosOriginalId/ontosEntityId; UI toggles render. Session: claude -r 37a1420d-86af-4380-bef3-0b74c6f8b4b9
) The batch importers created a contract from any uploaded file (and a product from any file on the products page), silently turning ODPS Data Products into Data Contracts because kind was never checked. Both upload views also only rendered failed items, so skips were invisible. - Add src/common/entity_kind.py: classify an entity as contract/product via kind (primary) + exclusive structural markers; ambiguous stays unknown. - create_contracts_from_files / create_products_from_files skip cross-type entities as 'skipped' items with a message pointing to the correct page. - Both upload views surface skipped (not just failed) in the import summary. - Tests for the classifier and cross-type/mixed-batch import. Folded into #860 (batch import). Tracks #929. Session: claude -r 171078d5-61c9-42b9-aeb6-5d378c386f35
Drop the trailing slash so the pattern matches a node_modules symlink (a file to git) as well as a real directory. The per-worktree node_modules symlinks used in local dev otherwise slip past 'node_modules/' and can be staged by git add -A. Session: claude -r 171078d5-61c9-42b9-aeb6-5d378c386f35
…es (#929) A README (or any free-form text) dropped on the contract uploader was turned into a junk contract: parse_uploaded_entities wrapped unstructured content into a minimal contract, and the kind guard only rejected positively-identified products, letting 'unknown' entities through. - parse_uploaded_entities now accepts only .yaml/.yml/.json structured payloads; an unsupported type or non-mapping/array raises (reported as a failed item). - Both batch importers now require a positively recognized entity: a contract must classify as 'contract', a product as 'product'. Cross-type files are skipped with a page hint; unrecognized files are skipped with a clear reason. - Tests: README/text rejection, structured-but-unrecognized skip, cross-type; updated the integration test that asserted the old lenient text fallback. Session: claude -r 171078d5-61c9-42b9-aeb6-5d378c386f35
The Data Contracts and Data Products upload dialogs had drifted apart: one had a drag-&-drop zone + paste box, the other only a 'Choose file(s)' button and a subtitle. Extract a single <ImportEntityDialog> used by both so they are identical: subtitle, the three import toggles, a multi-file drag-&-drop zone, and a paste-JSON box. - New components/common/import-entity-dialog.tsx (dropzone multiple:true). - Both views pass their own i18n labels, toggle state, and upload/paste handlers; data-products gains multi-file selection + drag-drop + paste. - Tighten accept to .json/.yaml/.yml (drop the stale 'plain text' affordance, matching the server-side non-ODCS/ODPS rejection). Session: claude -r 171078d5-61c9-42b9-aeb6-5d378c386f35
larsgeorge-db
added
the
stakeholder/fpiz
larsgeorge-db
added
the
stakeholder/fpiz
The dropzone advertised 'plain text' and 'a contract file' (singular); update the en strings to 'JSON or YAML (ODCS)' and 'file(s)', add the dialog subtitle key, and drop 'text files' from the upload tooltip — consistent with the shared ImportEntityDialog and the server-side non-ODCS rejection. Session: claude -r 171078d5-61c9-42b9-aeb6-5d378c386f35
mvkonchits-db
left a comment
mvkonchits-db
left a comment
Contributor
There was a problem hiding this comment.
Comment. The shared dialog direction is good, but this PR has structural problems. Items marked (verified) I confirmed in the commit list or diff.
- (verified) Undeclared stack. The branch contains #860's first commit, plus #862 and #863, but targets
development, and it isBEHIND. It is missing #860's later fixes (80f30dd56cross-type guard,6fae48e23reject non-ODCS/ODPS files). Without them, dropping an ODPS file into the contract dialog (or the reverse) creates a malformed entity from fallback fields. Please retarget tofeat/import-id-adoptor rebase once the stack merges, and expect add/add conflicts in the result model and types. - (verified) i18n: only
en/data-contracts.jsonis touched. The new shared-dialog strings are missing from de/es/fr/it/ja/nl, and the product-dialog keys are missing even fromen. - (reported, consistent with the diff) Closing and reopening only changes
open. Paste text, errors and the toggles (create domains, ID adoption, duplicate handling) are not reset, so a later import silently keeps earlier settings. - (reported) Pasted JSON is cleared regardless of outcome, because
uploadProductFiles()catches failures and returns normally, so the user loses the input they would need to retry. - No component test covers file selection, loading and error states, or close and reopen.
…jection build_odcs_from_db accepts a db_session but the semantic-assignment injection block ignored it and opened a second session via get_db_session(), which commits on exit. Under the test harness's shared single in-memory SQLite connection that commit lands on the outer per-test transaction, defeating its rollback and leaking committed contracts/products across tests (inflating the global-count assertions in test_version_family once the import round-trip tests started exercising the export path). In production it was a redundant session + commit on every export. Reuse the passed session when present (read-only lookups need no own session); fall back to get_db_session() only when no session was provided. Session: claude -r 171078d5-61c9-42b9-aeb6-5d378c386f35
…eat/import-id-adopt
…import-dialog-consistency
…ors (#855 review) Addresses mvkonchits-db's review on #860: 1. Cap uploads before parsing: new src/common/upload_limits.read_uploads_capped enforces max file count, per-file bytes and total bytes (413 on breach), reading one byte past the per-file cap so an oversize/alias-heavy payload cannot be loaded fully into memory. Both upload routes use it. 2. Decode UTF-8 per file in the contract route (not up front): a single non-UTF-8 file is now a failed item, no longer a 500 that aborts the batch. 3. Stop echoing raw exception text to clients — the two contract 500 details, the product 500 detail, and the per-entity create-failure item messages now return a generic message and log the full detail. (Parse/format messages, which describe the user's own file and carry no server internals, are kept.) 4. An empty ODPS array is a failed item (parity with the contract path), not a silent "0 imported successfully". Tests: unit coverage for the caps helper (count/per-file/total 413) and the product empty-array parity; route tests for non-UTF-8 -> failed item (not 500) and the file-count 413. Session: claude -r 171078d5-61c9-42b9-aeb6-5d378c386f35
…rd creator (#851 review) Addresses mvkonchits-db's must-fix on #862 (the authorization issue) plus the case/whitespace name-match should-fix: - The "create missing domains" toggle now requires data-domains WRITE on top of the entity's own write permission, enforced at all three import endpoints (contract upload, contract paste-import, product upload) via a new reusable user_has_feature_level() helper. Without it a contract/product writer could create top-level domains they cannot otherwise manage; now a clear 403. - Auto-created domains record the real uploader instead of the repo's "system" fallback: DataDomainCreate gains an optional created_by, threaded through _create_domain. - Name reconciliation is now trimmed and case-insensitive (data_domain_repo.get_by_name_normalized), so " finance " matches "Finance" rather than being left unassigned or creating a near-duplicate under auto-create. Also merges #860 (upload hardening + the cross-type/non-ODCS guards) so this branch is a proper superset of its base. Deferred to a follow-up (reviewer "should fix", not blocking): stale ontosDomainId list name-fallback (#851.2) and clearing a stale ontosDomainId on domain-less export (#851.3). Tests: real-creator attribution and trimmed/case-insensitive match (manager-level). Session: claude -r 171078d5-61c9-42b9-aeb6-5d378c386f35
…eat/import-id-adopt # Conflicts: # src/backend/src/controller/data_contracts_manager.py # src/backend/src/controller/data_products_manager.py # src/frontend/src/views/data-products.tsx
Adds a new per-role boolean privilege shown in the role editor Privileges tab under a new "Import Privileges" section, as the explicit app-role gate for adopting an entity's UUID from an imported file as the primary key. The gate itself (#853.1) replaces the previous default-on-for-any-writer behavior. - DB: AppRoleDb.can_adopt_entity_ids Boolean (default false, server_default 'false'); migration b1_add_adopt_entity_ids descends from n1_search_documents. - Pydantic: AppRoleBase/AppRoleUpdate/AppRole gain can_adopt_entity_ids; settings_manager._map_db_to_api surfaces it (repo write path serializes booleans unchanged via model_dump). - AuthorizationManager.user_can_adopt_entity_ids(groups): mirrors is_user_ontos_admin, resolves roles by case-insensitive group intersection. - Frontend: new "Import Privileges" section in the Privileges tab of role-form-dialog, mirroring Admin Role; TS AppRole gains the field; i18n keys added to all 8 locales (en + de/es/fr/it/ja/nl/pt). The enforcement at the import routes (and the flip of adopt_ids default to false) lands in the next commit. Co-authored-by: Isaac <no-reply@databricks.com>
…dation (#853 review) Addresses mvkonchits-db's must-fix items on #863: 1. ID squatting (#853.1): adopt_ids defaults to False on all three import endpoints. Even when explicitly on, adoption requires either the new per-role `can_adopt_entity_ids` privilege (prior commit) or feature ADMIN on the target entity. Non-privileged callers silently fall back to a generated UUID with the file id preserved as ontosOriginalId provenance — no hard 403 so unattended tooling degrades gracefully. 2. Canonicalize UUIDs before dup lookup and storage (#853.2): a new _canonical_uuid() parses via uuid.UUID (accepting upper/braced/urn/ hyphenless spellings) and returns the canonical lowercase hyphenated form. All three dup-check sites + both adoption sites use it, so a spelling variant can no longer bypass the duplicate check or create a second identity for the same logical UUID. 3. Validate on_duplicate at the route (#853.3): must be exactly 'skip' or 'new', otherwise 400. A typo no longer falls through to the "new" path and silently duplicates. 4. Skipped-dup existence oracle (#853.5): the batch result for a skipped duplicate now only surfaces the existing row's id/name when the caller is a feature-admin (new reveal_dup_detail flag computed at the route). Non-admin callers get a generic "Supplied id cannot be adopted" skip without entity_id/name, so a sibling import can't be used to probe for invisible entities. 5. Intra-batch reference remap for on_duplicate=new (#853.4): documented as vacuous given current architecture (ODCS carries no intra-file cross-refs; ODPS's only cross-ref inputPort.contractId points at contracts, which are imported through a separate endpoint). Scaffold hook noted for a future cross-entity batch flow. Tests: canonicalization (UPPERCASE + braced spelling), redaction for non-admin on both products and contracts, retained reveal-dup-detail behavior for admin callers. Co-authored-by: Isaac <no-reply@databricks.com>
…import-dialog-consistency # Conflicts: # src/frontend/src/views/data-contracts.tsx
…aste on failure + component test (#930 review) Addresses mvkonchits-db's review on #930 (shared ImportEntityDialog): - en/data-products.json: add the dialog i18n keys the products view was passing (dialogTitle, dialogDescription, createMissingDomains, createMissingDomainsHint, adoptIds, adoptIdsHint, duplicatesAsNew, duplicatesAsNewHint, dropActive, dropInactive, supportedFormats, orPasteOdps, pasteOdpsPlaceholder, importJsonButton, partialSuccess) — previously they only resolved via the hard-coded second-arg fallback. (The other 6 locales still need parity; defer to the #471 i18n sweep so this PR stays focused.) - Reset dialog state when the dialog opens: both views now zero createMissingDomains, adoptIds, duplicatesAsNew, the paste box, and any lingering error whenever openUploadDialog transitions true, so a later import doesn't silently inherit earlier settings. - Default adoptIds to false on the UI, matching the backend-level privilege gate (#853 review). - Keep pasted JSON on partial-success/failure: uploadProductFiles now returns a boolean (allCreated) and importPastedOdps only clears the textarea on full success; the contract-side importPastedOdcs no longer clears on partial. The user keeps the input they need to fix the failing entries and retry. - Component test for the shared dialog: label rendering, optional paste section, paste-submit wiring, uploading indicator. File-dropzone interactions are left to Playwright per the team-dialog precedent (jsdom + react-dropzone is unreliable). Co-authored-by: Isaac <no-reply@databricks.com>
…ute (#862 re-review) The previous commit (4610d15, data-domains gate) silently dropped the `file_inputs = [(name, raw) for name, raw, _ in capped]` line that #860 had introduced to feed `create_products_from_files`. Every product upload then raised `NameError` inside the generic `except Exception`, surfacing as a generic 500. A PR whose base isn't `development` only runs the title check, so backend tests never flagged it. Changes: - Restore the `file_inputs` build-up right after `read_uploads_capped()`. - Add two route-level tests: a golden-path product upload (guards the NameError regression) and a 403 for `create_missing_domains=True` when the user lacks `data-domains` write. - Expand the `user_has_feature_level` docstring to call out that it resolves group-based permissions only (no team/applied-role override) and fails closed on resolution error. Follow-up tracker filed as #944 (stale ontosDomainId on re-import + clearing a stale id on export). Session: claude -r 171078d5-61c9-42b9-aeb6-5d378c386f35 Co-authored-by: Isaac <no-reply@databricks.com>
…review) # Conflicts: # src/backend/src/routes/data_product_routes.py
…icts post #860/#865/#867) # Conflicts: # src/backend/src/controller/data_contracts_manager.py # src/backend/src/controller/data_products_manager.py # src/backend/src/routes/data_contracts_routes.py # src/backend/src/routes/data_product_routes.py # src/frontend/src/views/data-contracts.tsx # src/frontend/src/views/data-products.tsx
mvkonchits-db
approved these changes
Oct 9, 2026
mvkonchits-db
left a comment
mvkonchits-db
left a comment
Contributor
There was a problem hiding this comment.
Re-reviewed 2cbd0d63. Approving. My earlier items are addressed, and I checked them in the code.
- The branch now has
developmentmerged in. The PR diff is six frontend files (+591/-348) and the full CI suite passes. I also compared the tree against a rebase I did locally and it is identical, so nothing hidden came in with the merge. - Wrong-type files no longer create junk entities, because #860's guard is on
development. - Both views reset the toggles, paste text and errors when the dialog opens, and
adoptIdsdefaults to off. - The
enkeys for the products dialog are all present now (10 were missing ondevelopment, 0 now). - Pasted text is kept unless every entity was created, so a failed import keeps the input.
- There is a component test for the dialog.
Non-blocking notes:
- The other seven locales are still missing the new dialog keys. #471 is closed, so I filed #946 to track it. They fall back to
enin the meantime. - Retrying after a partial success re-submits entities that were already created. I did not run it, but I found no name-uniqueness guard on import, so with
adopt_idsoff they would likely get new UUIDs. That follows from keeping the pasted text, which I asked for. Keeping the text only when nothing was created would avoid it. - The products dialog shows failures only as a 10 second toast, because its error alert is on the page behind the modal. The contracts view passes an
alertinto the dialog. Passing the products error the same way would make the two consistent. - Opening the products dialog also clears a page-level load error, since
erroris shared state. - Six contract keys (
createMissingDomains,adoptIds,duplicatesAsNewand their hints) are missing fromen/data-contracts.json. They were already missing ondevelopmentand render from inline defaults. They are included in #946.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The Data Contracts and Data Products upload dialogs had drifted apart:
This extracts a single
components/common/import-entity-dialog.tsxused by both pages so they are identical: subtitle, the three import toggles (create-missing-domains / adopt-ids / duplicates-as-new), a multi-file drag-&-drop zone, and a paste-JSON box.accepttightened to.json/.yaml/.yml; the stale "plain text" affordance is removed to match the server-side non-ODCS/ODPS rejection.This stacks on the import PRs and must merge after them (the toggles come from #862, multi-file +
BatchImportResultfrom #860, id-adoption from #863):feat/import-batch→ developmentfeat/import-domain-recon→ feat(import): single + batch (array-in-file + multi-file) upload for ODCS & ODPS (#855) #860feat/import-id-adopt→ feat(import): domain reconciliation + ontos* round-trip on ODCS/ODPS import (#851) #862 ← this PR is branched off feat(import): adopt YAML entity UUID as primary key + duplicate handling (#853) #863Until #860/#862/#863 land, this PR will not apply cleanly to
development. Related correctness work for the uploaders is tracked in #929.Verification
tsc --noEmitclean. Visual/manual upload testing to follow on the integrated branch.