feat(namespace): support Lance Namespace across table operations by FANNG1 · Pull Request #35 · daft-engine/daft-lance

FANNG1 · 2026-06-19T10:02:17Z

Closes #50 (motivating use case: Gravitino Lance REST server managed tables + Daft multimodal processing).

Summary

add a shared Lance namespace resolver for URI-or-namespace validation, namespace client construction, table resolution, and storage option merging
wire namespace table support through daft_lance.read_lance, the new daft_lance.write_lance, LanceDataSink, merge/index/compaction operations, and worker scan reconstruction
keep dataset-open state in an explicit LanceDatasetHandle instead of attaching Daft-private attributes to LanceDataset; retain the historical construct_lance_dataset() return type through a compatibility wrapper
explicitly thread resolved URIs, vended storage options, and namespace commit kwargs through scan, merge, and distributed index paths
add dir namespace coverage for reads/writes, transform and slow-path merges, segmented/non-segmented distributed indexes, compaction, credential vending, and worker reconstruction, plus a gated Gravitino Lance REST namespace integration test
tighten namespace table-not-found handling and request vended credentials explicitly

Validation

.venv/bin/pytest -q (324 passed, 5 skipped, 2 xfailed, 2 xpassed)
.venv/bin/ruff format --check daft_lance tests
.venv/bin/ruff check daft_lance tests
.venv/bin/mypy daft_lance
git diff --check

Cross-repo plan

daft_lance.write_lance in this PR looks like a copy of DataFrame.write_lance, but the
direction is the opposite: Eventual-Inc/Daft#6957 already migrated all of daft.io.lance to
this package, and DataFrame.write_lance's inline body is the last piece left behind
upstream. Eventual-Inc/Daft#7282 proposes reducing it to a thin delegation to
daft_lance.write_lance, making this module the single canonical implementation.

Until Eventual-Inc/Daft#7282 lands, namespace users call daft_lance.read_lance /
daft_lance.write_lance directly. (An earlier revision of this PR shipped a patch_daft()
monkeypatch as interim glue for Daft's native entry points; it was removed in favor of the
direct entry points since it had never been released.)

…tion Build on the initial namespace read/write support: - Extend namespace params (namespace_impl/namespace_properties/table_id) to merge_columns, merge_columns_df, create_scalar_index, compact_files, threading namespace kwargs into their commit sites. - Add an explicit daft_lance.write_lance() and make the daft monkeypatch opt-in via patch_daft(); wire write_lance(mode="merge") through namespace. - Fix _declare_table fallback that always raised ImportError on the CreateEmptyTableRequest import; keep only declare_table. - Narrow overwrite resolution to only declare on TableNotFound instead of swallowing every describe_table exception. - Drop dead code (open_lance_dataset, pylance<5 compat) and dedupe the worker dataset-reopen logic into open_dataset_from_open_kwargs. - Cover all entry points with dir-namespace tests plus a Gravitino REST namespace e2e roundtrip. - Work around an upstream daft+lance native teardown SIGSEGV (unrelated to test outcomes) via a pytest_unconfigure hard-exit after reporting.

…nk.start() Constructing a LanceDataSink no longer talks to the namespace or opens the dataset; all resolution moves to start(), which Daft runs once on the driver before the sink is serialized to workers. This keeps declare_table from firing during plan construction and from leaving orphan declared tables behind when local parameter validation fails. Create-mode resolution is now describe-first (check_declared=True): a real existing table fails fast with a clear error suggesting overwrite/append, a declared-only stub is reused as a placeholder (write_fragments switches to overwrite when the stub materialized a dataset), and a lost declare race is recovered by re-describing and classifying. resolve_namespace_table returns a ResolvedNamespaceTable carrying uri/storage_options/placeholder state so the sink does not have to re-describe. Claude-Session: https://claude.ai/code/session_01VyD31uRyBSEmKvPQhLtM5E

…locations Previously io_config was converted to storage options only when a plain uri was passed, so a namespace that resolves to s3://... without vending credentials silently ignored the user's io_config. construct_lance_dataset and LanceDataSink now derive storage options from io_config against the resolved location and layer them as: io_config-derived < user-provided storage_options < namespace-vended. Plain-uri behavior is unchanged (user-provided options still replace io_config-derived ones). Claude-Session: https://claude.ai/code/session_01VyD31uRyBSEmKvPQhLtM5E

daft_lance/ sources and the namespace test files now pass mypy strict. The bulk came from LanceDataSink._namespace_kwargs being annotated dict[str, object], which poisoned every **-expansion into typed lance APIs. Also widens read_lance's default_scan_options to dict[str, Any] to match the documented usage ({"with_row_address": True}) and the other entry points. Pre-existing errors in the rest of tests/ are out of scope for this branch. Claude-Session: https://claude.ai/code/session_01VyD31uRyBSEmKvPQhLtM5E

…phan properties - read_lance / merge_columns(_df) / create_scalar_index / compact_files docstrings now describe table_id / namespace_impl / namespace_properties - namespace_properties without namespace_impl now fails fast instead of being silently ignored - README documents io_config fallback for namespace locations and the DAFT_LANCE_NAMESPACE_CACHE_SIZE environment variable Claude-Session: https://claude.ai/code/session_01VyD31uRyBSEmKvPQhLtM5E

- overwrite-mode resolution now describes with check_declared=True (initial and declare-race recovery): strict namespace impls 404 a plain describe on declared-only stubs, so the previous recovery path failed in exactly the case it existed for. A declared-only stub is a valid overwrite target. - mode=create over a namespace placeholder that materialized a dataset is rejected with a clear error when use_mem_wal=True; the MemWAL path cannot apply the overwrite-placeholder semantics. - plain-uri entry points treat storage_options={} as unset again (falsy fallthrough to io_config-derived options), matching pre-branch behavior. Claude-Session: https://claude.ai/code/session_01VyD31uRyBSEmKvPQhLtM5E

…nt count - pickle a started sink and run write()/finalize() on the copy, codifying the driver-resolves/worker-writes contract instead of relying on manual verification - compact_files test now asserts the fragment count actually shrinks, not just data correctness Claude-Session: https://claude.ai/code/session_01VyD31uRyBSEmKvPQhLtM5E

… storage_options handling - describe_table/declare_table now pass vend_credentials=True: per the lance-namespace spec, whether credentials are returned is implementation-defined when the flag is unset, so the documented "namespace vends storage_options" behavior was not guaranteed - LanceDataSink now treats storage_options={} the same as the read entry points (falsy fallthrough to io_config-derived options); previously write_lance dropped io_config credentials for that input while read_lance kept them Claude-Session: https://claude.ai/code/session_01VyD31uRyBSEmKvPQhLtM5E

The old message predated namespace support and directed users to "the daft-lance package" — which is where they already are. Show the actual read_lance(namespace_impl="rest", ...) usage instead. Claude-Session: https://claude.ai/code/session_01VyD31uRyBSEmKvPQhLtM5E

Module docstring covers the three load-bearing decisions (serialize the triple + per-process client cache, describe-first resolution, defensive error classification), and resolve_namespace_table documents the full create/overwrite/read state machine including the declare-race legs. Claude-Session: https://claude.ai/code/session_01VyD31uRyBSEmKvPQhLtM5E

patch_daft() was introduced on this branch and never released, so it can be removed without a deprecation cycle. Until Eventual-Inc/Daft#7282 delegates DataFrame.write_lance / read_lance natively, namespace users call daft_lance.read_lance / daft_lance.write_lance directly — which the README documents as the primary usage anyway. This also removes the only monkeypatch in the package and avoids having to keep the patch compatible with the upstream signature change coming in #7282. Claude-Session: https://claude.ai/code/session_01VyD31uRyBSEmKvPQhLtM5E

Claude-Session: https://claude.ai/code/session_01VyD31uRyBSEmKvPQhLtM5E

universalmind303 · 2026-07-21T16:07:03Z

thanks for the PR @FANNG1, Will take a look asap!

universalmind303

I ran this through a pretty deep review & a no-mocks e2e harness (real dir namespace via lance-namespace: create/append/read/pushdowns/merge columns/index/compaction all through the triple, cross-checked with plain pylance). happy path holds up nicely. found a handful of real issues though.

I think the percent-encoding & compat wrapper ones need fixing before merge, the rest can be follow ups.

universalmind303 · 2026-07-21T16:43:06Z

+    """
+    parsed = urlparse(location)
+    if parsed.scheme == "file":
+        return parsed.path


urlparse().path doesnt percent-decode, & the dir namespace's declare_table vends encoded uris (file:///path/with%20space/t.lance) while describe_table returns plain paths. create (& overwrite on a missing table) resolves through the declare response, so any root or table name with a space or non-ascii char writes to a literal with%20space sibling dir: the write reports success with stats, the registered location stays empty, & the next read_lance fails with Dataset ... was not found. reproduced end to end on this branch.

Suggested change

return parsed.path

return unquote(parsed.path)

(with unquote added to the urllib.parse import)

universalmind303 · 2026-07-21T16:43:06Z

+    )
+
+
+def construct_lance_dataset(


the compat wrapper keeps the return type but not the behavior: main stashed _lance_open_kwargs / _daft_default_scan_options on the returned dataset & LanceDBScanOperator read them back when building scan tasks. on this branch LanceDBScanOperator(construct_lance_dataset(uri, storage_options=...)) plans fine on the driver but ships empty open_kwargs to workers (s3 auth failures mid scan), & default_scan_options={"nearest": ...} is silently dropped so a vector search degrades to a full scan returning all rows. verified empirically: the same script returns the nearest opts on main & None here.

i think we should either restore the stashing, or drop construct_lance_dataset entirely & document the break. a wrapper that type-checks but silently behaves differently is kinda the worst of both

universalmind303 · 2026-07-21T16:43:06Z

                storage_options=self._storage_options,
                data_storage_version=self._data_storage_version,
                use_legacy_format=self._use_legacy_format,
+                **self._namespace_kwargs,


mem-wal + namespace never works: start() already declared the table, & pylance's write_dataset(mode="create", namespace_client=...) unconditionally declares again, so the first worker write always dies with

RuntimeError: Exception occurred while writing to Lance Write: TableAlreadyExistsError: Table already exists: t

(same for overwrite on a not-yet-materialized table). there's no coverage for the combo either. if fixing it here is too much, I'd be fine with an explicit unsupported error & a follow up.

universalmind303 · 2026-07-21T16:43:06Z

@@ -566,8 +627,69 @@ def compact_files(
    )

    return compact_files_internal(


the scan path re-resolves per task via open_dataset_from_open_kwargs so workers get fresh vended creds, but compact_files / create_scalar_index / merge_columns pickle the live dataset into the worker udfs & pylance's __setstate__ nulls _namespace_client. workers end up on the plan-time storage_options snapshot with no refresh, so against a catalog vending short-lived sts creds (the gravitino case from #50) a long compaction/index/merge dies with auth errors partway through while the equivalent scan keeps working.

related: the worker-side reopen never asks for vend_credentials either (pylance's internal describe doesnt set it), while the driver-side _describe_table sets it explicitly for exactly that reason. couldnt we route the distributed ops through the same triple-based reopen the scan path uses?

universalmind303 · 2026-07-21T16:43:06Z

+        is the single place where namespace side effects (``declare_table``) and
+        dataset opens happen.
+        """
+        resolved = self._resolve_table()


if start() declares the table & anything after fails (stale files at the vended location, worker crash, storage error), the declared entry never gets cleaned up. there's no drop_table anywhere in daft_lance, so retrying the same create is permanently wedged on TableAlreadyExistsError until someone drops the table by hand. should we compensate with a drop_table when a create fails after declaring? (only for tables this write declared, to not race other writers)

FANNG1 marked this pull request as draft June 19, 2026 12:17

This was referenced Jul 16, 2026

Support Lance Namespace table addressing (catalog-managed Lance tables, e.g. Gravitino Lance REST server) #50

Open

Complete the daft.io.lance -> daft_lance migration: delegate DataFrame.write_lance (follow-up to #6957) Eventual-Inc/Daft#7282

Open

FANNG1 changed the title ~~Add Lance namespace read/write support~~ feat(namespace): support Lance Namespace across table operations Jul 16, 2026

fanng added 13 commits July 16, 2026 19:02

feat: add lance namespace read write support

403e4b6

fix(namespace): tighten missing table handling

7b1a7c4

FANNG1 force-pushed the namespace branch from 13ad2b5 to 3bf5943 Compare July 16, 2026 10:08

FANNG1 marked this pull request as ready for review July 16, 2026 10:10

style: format scalar index namespace reload

fc7cda3

FANNG1 force-pushed the namespace branch from 37cee25 to fc7cda3 Compare July 16, 2026 13:38

chore: drop unused __future__ import from package init

6f3bd94

Claude-Session: https://claude.ai/code/session_01VyD31uRyBSEmKvPQhLtM5E

FANNG1 marked this pull request as draft July 16, 2026 13:54

refactor(namespace): remove legacy and unrelated compatibility paths

ea3293d

FANNG1 mentioned this pull request Jul 21, 2026

[Epic] Production-ready Lance Namespace support across daft-lance and Daft #51

Open

5 tasks

fanng added 3 commits July 21, 2026 11:36

refactor(namespace): make Lance open context explicit

fe11b9a

fix(namespace): propagate managed versioning through commits

f7f665c

refactor(namespace): tighten integration types

2b97eb1

FANNG1 marked this pull request as ready for review July 21, 2026 03:50

universalmind303 self-requested a review July 21, 2026 15:14

universalmind303 self-assigned this Jul 21, 2026

universalmind303 reviewed Jul 21, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(namespace): support Lance Namespace across table operations#35

feat(namespace): support Lance Namespace across table operations#35
FANNG1 wants to merge 19 commits into
daft-engine:mainfrom
FANNG1:namespace

FANNG1 commented Jun 19, 2026 •

edited

Loading

Uh oh!

universalmind303 commented Jul 21, 2026

Uh oh!

universalmind303 left a comment •

edited

Loading

Uh oh!

universalmind303 Jul 21, 2026 •

edited

Loading

Uh oh!

universalmind303 Jul 21, 2026 •

edited

Loading

Uh oh!

universalmind303 Jul 21, 2026

Uh oh!

universalmind303 Jul 21, 2026

Uh oh!

universalmind303 Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -566,8 +627,69 @@ def compact_files(
		)

		return compact_files_internal(

Uh oh!

Conversation

FANNG1 commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Cross-repo plan

Uh oh!

universalmind303 commented Jul 21, 2026

Uh oh!

universalmind303 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

universalmind303 Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

universalmind303 Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

universalmind303 Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

universalmind303 Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

universalmind303 Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

FANNG1 commented Jun 19, 2026 •

edited

Loading

universalmind303 left a comment •

edited

Loading

universalmind303 Jul 21, 2026 •

edited

Loading

universalmind303 Jul 21, 2026 •

edited

Loading