A collection of shared utilities, libraries, and command-line tools that form the technical backbone of the CELINE data platform. Provides reusable building blocks for data pipelines, governance, lineage, metadata management, and platform integrations.
Not an end-user application — a platform utility layer embedded into CELINE applications and executed within orchestrated environments using Meltano, dbt, Prefect, and OpenLineage.
- Centralise cross-cutting platform logic used by multiple CELINE projects
- Provide opinionated but extensible tooling for data pipelines
- Enforce consistent governance and lineage semantics
- Reduce duplication across pipeline applications
- Act as a stable foundation for CELINE-compatible services and workflows
A declarative governance.yaml specification defines the metadata, access control, and dataspace exposure rules for each dataset.
The GovernanceRule model covers:
- Dataset ownership (
ownership,attribution), resolved through an owner registry - License and access level (
open,internal,restricted—secretis accepted for compatibility and normalised torestricted) - Data classification (
pii,green,yellow,red) and retention - Tags, documentation links, and source system
row_filters— list of filter specs ([{handler, args}]) for per-subject consent-based row filtering- Two exposure gates, ANDed:
exposecontrols whether the dataset is listed in the DCAT catalogue and served by the API, whiledataspace.exposecontrols whether it is offered into the dataspace as an EDC asset.exposeis tri-state — unset falls back todataspace.exposeso pre-split files keep their behaviour
Extended blocks for DCAT-AP 3.0 and dataspace integration:
dcat: block — propagated to the DCAT-AP catalogue by dataset-api:
publisher_uri— overrides the API-level fallback publisherthemes— EU Publications Office data-theme URIslanguage_uris— dct:language URIsspatial_uris— dct:spatial URIsaccrual_periodicity— dct:accrualPeriodicity URIconforms_to— dct:conformsTo URItemporal.start/temporal.end— dct:temporal coverage
dataspace: block — consumed when registering datasets in EDC:
expose— offer the dataset into the dataspacecontract_required— enablesds:contractRequiredODRL constraintconsent_required— enablesds:consentStatusODRL constraint and consent-based row filteringodrl_action— default ODRL action (defaultuse)purpose— ODRL purpose valuesmedallion— data quality level (gold / silver / bronze)
ontology: block — which mapping spec says what the columns mean: spec (a shared mapping published in celine-ontologies) or spec_file (a path relative to the governance file). Exactly one, enforced by the schema.
Governance rules are resolved with pattern matching via GovernanceResolver: exact key first, then the longest matching glob, then defaults alone. The chosen rule is overlaid on the defaults using the fields the file explicitly set — not truthiness — so expose: false withdraws a dataset instead of being silently dropped. tags and dataspace.purpose union, consent_required and contract_required OR, ownership / row_filters / ontology replace wholesale. See the format reference for the full table.
celine.governance is the single parser: dataset-api, ds and celine-superset import it rather than reimplementing it. EDC-specific sub-objects in the dataspace: block belong to ds and are ignored here via model_config = ConfigDict(extra="ignore") rather than rejected.
Structured execution layer for:
- Meltano ingestion pipelines
- dbt transformations, tests, and seeds (
dbt_seedwrapper) - Prefect-based Python flows (Prefect 3.x)
The PipelineRunner coordinates execution, logging, error handling, and lineage emission consistently across tools. Pipeline run environment variables can be injected via pipeline_run_envs for local runtime configuration.
See the pipeline tutorial.
- Automatic emission of START, COMPLETE, FAIL, and ABORT events
- Dataset-level schema facets
- Data quality assertions from dbt tests
- Custom CELINE governance facets (including
row_filters,medallion,classification)
The DatasetClient enables:
- Schema and table introspection
- Column metadata inspection
- Safe query construction
- Export to Pandas
- MQTT pipeline run events, published through
celine-sdk - Keycloak client-credentials tokens, for authenticating to a protected Marquez
Keycloak and Superset administration was removed in 2.3.0 along with the
celine-utils admin command tree — it was a replicable-setup tool with no remaining
callers. Provisioning lives in celine-policies.
celine-utils governance generate marquez --app <app> # scaffold governance.yaml from Marquez
celine-utils pipeline init app <name> # scaffold a new pipeline app
celine-utils pipeline run (meltano | dbt | prefect) # run a pipeline stageFull reference: CLI.
src/celine/
governance/ the thin core — three dependencies, imported by other repositories
utils/
cli/
common/
datasets/
pipelines/
schema/ JSON Schemas — published, and symlinked into the package
docs/
tests/
integration-tests/
Environment-driven via pydantic-settings:
- Environment variables first
- Optional
.envfiles - Typed validation with container-friendly defaults
Two tracks — see the documentation index.
Governance as a library
- The governance library —
celine.governanceAPI, the three-dependency contract, merging, validation governance.yamlformat — the grammar, resolution, the two exposure gatesowners.yamlregistry — owner aliases to canonical identities- Schemas — the three published JSON Schemas
Building and running pipelines
- Pipeline Tutorial — end-to-end pipeline setup guide
- CLI — full CLI reference
- Environment — every variable and its default
Why choices were made — Decisions.
uv add celine-utils # governance only — three dependencies
uv add "celine-utils[pipelines]" # dbt / Meltano / Prefect / lineage
uv add "celine-utils[all]" # pipelines + the typed OpenLineage facetParsing governance.yaml needs no extras. That is deliberate and enforced by CI —
see ADR-0001.
- Data engineers
- Platform engineers
- CELINE application developers
Copyright © 2025 Spindox Labs
Licensed under the Apache License, Version 2.0.