Read enum on properties everywhere allowed values are used - #1569
Merged
Conversation
ODCS v3.2.0 declares allowed values as enum, a list of entries with a value and optional label, description, id, tags and more. One resolver in datacontract/model/enum_values.py reads it ahead of the three legacy representations (logicalTypeOptions.enum, the enum custom property, the invalidValues rule) and replaces the seven private copies in the exporters, the check builder and the dbt test mapping. The check builder and sodacl export skip the quality-rule source, which they turn into a check of their own. Importers from JSON Schema, Avro, Protobuf and DCS write enum entries instead of an invalidValues rule or custom properties. Pydantic export types enumerated strings and integers as typing.Literal, protobuf export numbers entries from their id, Avro export treats physicalType enum with values as an Avro enum, and HTML export lists the values with labels and descriptions.
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1560 (#1557). Targets the
odcs-3.2.0branch.ODCS v3.2.0 (RFC 0033) declares allowed values as
enum, a list of entries with avalueand optionallabel,description,id,tags,customProperties,authoritativeDefinitions.One resolver
datacontract/model/enum_values.pyreadsenumfirst, then the legacy representations in order:logicalTypeOptions.enum, theenumcustom property (list or JSON string, from DCS), theinvalidValuesquality rule. It replaces seven private copies in the avro, jsonschema, great-expectations, data-caterer and sodacl exporters, the check builder and the dbt test mapping. The check builder and sodacl export passinclude_quality_rule=False, because they turn the quality rule into a check of its own and would otherwise check it twice.Producers
datacontract importfrom JSON Schema, Avro (including optional enums in unions, whose symbols were dropped before), Protobuf and DCS writesenumentries instead of aninvalidValuesrule oravroSymbols/enumValuescustom properties (enumValuesstays for the protobuf numbers).create_propertygained anenumargument.Consumers that can carry more than values
datacontract test:field_enumcheck from the entriestyping.Literal[...]enumblocks numbered from each entry'sidwhen numeric, else by positionphysicalType: enumand values exports as an Avro enum without needing theavroTypecustom propertyfield.enumfrom any representationTests
New
tests/test_enum_values.py(resolver priority, DuckDB end-to-end pass and fail, every exporter, JSON Schema and DCS import). Avro and protobuf import expectations updated to the entries. Full suite: 2323 passed.Docs:
docs/docs/schema.mdusesenumentries in its examples and notes thatlogicalTypeOptions.enumis still accepted.🤖 Generated with Claude Code
https://claude.ai/code/session_01C1xhK6DjRLWND1ntBcS8fG