Exclude runId from DatasetVersion hash to prevent version explosion

[kalra-mohit]

DatasetVersion is supposed to identify the content of a dataset at a point in time — namespace, source, physical name, schema, lifecycle state. But the hash that produces its UUID (Utils.newDatasetVersionFor(DatasetVersionData), the single private method backing all three public overloads used by OpenLineageDao, RunDao, and DatasetVersionDao) also folded in data.getRunId():

final byte[] bytes =
    VERSION_JOINER
        .join(
            data.getNamespace(),
            data.getSourceName(),
            data.getDatasetName(),
            data.getPhysicalName(),
            data.getSchemaLocation(),
            data.getFields().stream().map(Utils::joinField).collect(joining(VERSION_DELIM)),
            data.getLifecycleState(),
            data.getRunId())   // <-- shouldn't be here
        .getBytes(UTF_8);

Since runId is unique per run, every single run that writes to a dataset mints a brand new DatasetVersion — and new dataset_versions / dataset_versions_field_mapping rows — even when the dataset's content is byte-for-byte identical to the previous run. High-frequency jobs writing the same dataset over and over blow this table up, which is exactly the query timeouts described in #3082.

flowchart TD
    W["Run writes dataset X
(same schema/content as the previous run)"]
    W --> H1["Old hash input:
namespace + source + name + physicalName +
schema + fields + lifecycleState + runId"]
    H1 --> V1["Different UUID every run
dataset_versions row explosion"]
    W --> H2["New hash input:
namespace + source + name + physicalName +
schema + fields + lifecycleState"]
    H2 --> V2["Same UUID as previous run
ON CONFLICT(version) DO UPDATE fires"]
    V2 --> T["Trade-off: dataset_versions.run_uuid
flips to the most recent writer of that content"]

Loading

Fix: dropped data.getRunId() from the hash. It's now purely content-based, matching what the docs say DatasetVersion is, and matching the sibling method newDatasetSchemaVersionFor, which never included runId in the first place. Because newDatasetVersionFor(DatasetVersionData) is the single choke point for all three overloads, this fixes it everywhere a DatasetVersion gets minted — no call sites changed, runId is still accepted as a parameter and still recorded via dataset_versions.run_uuid, it's just out of the content hash now.

Testing: added UtilsTest#testDatasetVersionEqualAcrossDifferentRunsWithSameDatasetContent — builds two DatasetVersions from identical content but two different, randomly generated runIds and asserts they're equal. Fails pre-fix (two different UUIDs), passes post-fix. Full unit suite is green: ./gradlew :api:testUnit, 119 tests.

I wasn't able to run the Postgres-backed DAO/integration tests (RunDaoTest, DatasetDaoTest, etc.) locally — Testcontainers in my sandbox can't negotiate with the local Docker Engine (the bundled docker-java client defaults to API v1.32, the engine here needs >= v1.40). Not related to this change, just a local environment gap. Would appreciate CI running those.

One trade-off I want to flag explicitly

DatasetVersionDao.upsert(...) already has ON CONFLICT(version) DO UPDATE SET run_uuid = EXCLUDED.run_uuid, .... Before this fix, runId being in the hash guaranteed every write produced a unique version, so that conflict branch never actually fired for OpenLineage-ingested writes.

After this fix, two different runs writing identical content now collide on version and hit that DO UPDATE. Since dataset_versions has one row per distinct version, the update reassigns that row's run_uuid to whichever run most recently wrote the content.

That matters because RunDao's output_versions (surfaced via GET /runs/{id}) is essentially a one-to-one join on dv.run_uuid = r.uuid. So: Run A writes content X, its output_versions includes that version. Run B later writes the exact same content X — the shared row's run_uuid flips to Run B, and Run A's output_versions silently loses that dataset, even though Run A genuinely produced it at the time.

I think this is an acceptable trade-off — deduping identical content is the whole point of the fix, and keeping runId in the hash is the bug we're fixing here. It only affects "which run(s) produced this exact version" provenance for content that's genuinely unchanged across runs, not the correctness of the version itself or its schema/fields.

A complete fix would need dataset_versions.run_uuid to model a proper one-to-many relationship (one version legitimately "produced" by many runs that happened to write the same content) — something like a many-to-many run-to-dataset-version join table, mirroring how runs_input_mapping already works for inputs, plus updates to RunDao.output_versions and the API contract. That's bigger than this PR; happy to scope it as a follow-up if maintainers want it.

Question for maintainers: is the shrinking-output_versions-over-time behavior an acceptable trade-off given what this fixes, or would you rather pair this with a minimal many-to-many link now?

Fixes #3082.

[kalra-mohit]

@merobi-hub whenever you get a chance to take a look, happy to address any feedback!