MiniLake

Free, open-source local Databricks emulator for offline development and testing.

Real SQL & Spark execution · Unity Catalog hierarchy · Databricks SDK compatible · Terraform compatible · MIT licensed

Website · Documentation · GitHub · Container Image (GHCR)

---

MiniLake is a free, local Databricks API emulator — a single-developer tool for testing databricks-sdk/Terraform code against real SQL, real Delta Lake, and real Job execution, without paying for cloud compute.

Quick Start

# Option 1: PyPI
pip install minilake
minilake --port 8000

# Option 2: GitHub Container Registry
docker run -p 8000:8000 ghcr.io/dmux/minilake:latest

# Option 3: Clone and build
git clone https://github.com/dmux/minilake && cd minilake
docker compose up -d

# Verify (any option)
curl http://localhost:8000/_minilake/health

Then open http://localhost:8000/ui/ for the built-in SQL workspace — an Athena-style query editor with a data catalog, saved queries and query history (details).

No account, no API key, no sign-up — and the container image downloads nothing at runtime: DuckDB's delta extension and the Delta / Unity Catalog Spark jars are baked in at build time, so it works air-gapped (details).

Then point any Databricks client at it:

from databricks.sdk import WorkspaceClient

w = WorkspaceClient(host="http://localhost:8000", token="dev")
w.catalogs.create(name="vendas")

More in Getting Started.

Documentation

Getting Started	Install, first catalog and query, internal endpoints
Web UI	The built-in SQL workspace at /ui
Configuration	Every environment variable, persistence, HTTPS/TLS
Databricks SDK	Unity Catalog, warehouses, SQL and jobs from Python
Terraform & Asset Bundles	The provider, and bundle deploy / bundle run
Spark & Delta Lake	Real Delta files, real Spark jobs, spark.table() by name
MCP Server	67 tools for LLM agents — examples, tool reference, troubleshooting
Testing & development	Running the suite, adding an API group
Releases & CI/CD	How a tag becomes a published image
Feature status	Endpoint-by-endpoint status and design rationale
API coverage	Measured: every SDK endpoint probed through the real databricks CLI
Emulation roadmap	What is worth building next, with measured gap counts

Supported Services

Service	Status	Notes
Unity Catalog (catalogs, schemas, tables, volumes, functions)	✅ Real	Each catalog = its own DuckDB database (ATTACH), native catalog.schema.table addressing; a SQL function becomes a real DuckDB macro, callable by its three-part name
Unity Catalog metastore	✅ Real	One synthetic metastore: current-metastore-assignment, metastore_summary, list/get
EXTERNAL Delta Tables	✅ Real	Real Delta files; INSERT/UPDATE/DELETE via a generated Spark job, reads via delta_scan()
SQL Statement Execution	✅ Real	Real DuckDB; JSON_ARRAY/ARROW_STREAM/CSV, INLINE/EXTERNAL_LINKS; result manifest carries column types
SQL Warehouses	✅ Real	Full CRUD + lifecycle
Query History	✅ Real	w.query_history.list() over everything executed, failures included
Saved Queries	✅ Real	w.queries.* CRUD with update_mask
Alerts	✅ Real	w.alerts.* CRUD; conditions are really evaluated against query results, and back sql_task.alert
Web UI	✅ Real	Athena-style SQL workspace at /ui — see Web UI
Jobs	✅ Real	Sibling Docker container execution (Spark) or subprocess fallback; real DAG scheduling (depends_on/run_if); runs/submit one-shot runs; sql_task.file/.query/.alert all execute for real
Workspace	✅ Real	File-backed notebook/script storage; raw-bytes workspace-files sync powers databricks bundle deploy / bundle run
DBFS & Files API	✅ Real	File-backed storage, chunked upload
Secrets	✅ Real	Real CRUD; values only resolvable inside job env vars, never via direct API (matches real Databricks). Scope ACLs are stored and read back, never enforced
Clusters	✅ Real state machine	CRUD + timed lifecycle transitions, update/pin/unpin; no real Spark compute (by design)
Cluster policies & instance pools	✅ Real CRUD	Resolve and validate against clusters; never enforced — there is no compute to constrain
Permissions	✅ Real CRUD	Single-user "allow-all" default (by design — see Gaps)
UC Grants	✅ Real CRUD	w.grants.* and databricks_grants, with real privilege inheritance (a catalog grant is effective on its schemas and tables); never enforced
Workspace admin	✅ Real CRUD	Git credentials, IP access lists, global init scripts, notification destinations, instance profiles, workspace-conf — stored and read back, never enforced
Legacy SQL (preview/sql)	✅ Real	Queries, alerts, dashboards, widgets, visualizations, data sources — adapters over the modern stores, so both surfaces see one object
Identity & SCIM	✅ Real CRUD	Current user, plus Users/Groups/ServicePrincipals CRUD, SCIM PATCH and filtering. Identities are records, not credentials — see Gaps
Tokens	✅ Real	w.tokens.*; the value is returned once, as in the real API — but authenticates nothing
Persistence (MINILAKE_PERSIST=1)	✅ Real	JSON snapshot on shutdown, restored on startup
Unity Catalog protocol for Spark	✅ Real	spark.table("cat.sch.tbl") resolves against minilake — see Spark & Delta Lake
JupyterLab + PySpark + Delta (optional)	✅ Real	docker compose --profile notebook up
MCP Server (optional, MINILAKE_MCP=1)	✅ Real	67 tools + resources + prompts at /mcp — see MCP Server
Repos, multi-language notebooks, DBT/pipeline tasks, DLT, Model Registry, Vector Search, Lakeview/Genie, Delta Sharing	🚫 Not implemented	Returns 501 NOT_IMPLEMENTED. For the measured picture, see API coverage

Known Gaps

These are deliberate, not oversights — minilake targets one developer running it locally, not a shared or multi-tenant server:

• No real authentication — any token is accepted. SCIM identities and personal access tokens are records, not credentials: creating a user makes no way to sign in, and revoking a token locks nobody out.
• No access-control enforcement — the Permissions API is real CRUD but always allow-all, so a test that passes here says nothing about grants in a real workspace.
• No real Spark compute for Clusters — state machine only; real compute happens through Jobs' sibling containers instead.
• Single process, no HA — and DuckDB's single-writer model means concurrent load contends on locks.
• Uneven test coverage — jobs.py, sql_statements.py and unity_catalog.py are covered mostly on happy paths, not edge cases.

Contributing

See CONTRIBUTING.md for the project structure, how to add a new API group, and the PR checklist.

License

MIT — see LICENSE.