Raster index and layers

README.md

@@ -18,6 +18,10 @@ A monorepo containing Python-based tools and libraries for Earth data projects.
 - `macrostrat.dinosaur`: Utilities for on-the-fly database migration and
   conformance testing
 - `macrostrat.package_tools`: Monorepo versioning and PyPI publishing utilities
+- `macrostrat.raster_index`: An index of cloud-optimized rasters (COGs), grouped
+  into named layers, with the `raster_layers` schema and a registration CLI
+- `macrostrat.raster_layers`: Serves those layers as mosaicked map tiles, as
+  FastAPI routes mountable in any application
 - `macrostrat.utils`: Helpers for logging and command-line apps
 
 ## Development

dinosaur/CHANGELOG.md

@@ -1,5 +1,10 @@
 # Changelog
 
+## [4.2.1] - 2026-08-12
+
+- Fix `database_cluster` on testcontainers 4.x, which now sets `tmpfs` itself and
+  errored on the value passed through `with_kwargs`
+
 ## [4.2.0] - 2026-06-19
 
 - Rework `upgrade_cluster` function and remove legacy `upgrade_cluster_legacy`

dinosaur/macrostrat/dinosaur/cluster.py

@@ -99,11 +99,16 @@ def database_cluster(
         in_memory = True
 
     if in_memory:
-        container.with_kwargs(
-            tmpfs={
-                "/var/lib/postgresql/data": "uid=999,gid=999,mode=0700",
-            }
-        )
+        # testcontainers >= 4.x owns `tmpfs` itself and passes it to Docker
+        # explicitly, so smuggling it through `with_kwargs` collides with that
+        # argument ("got multiple values for keyword argument 'tmpfs'"). Use the
+        # dedicated mount API where it exists, and fall back for older versions.
+        data_dir = "/var/lib/postgresql/data"
+        options = "uid=999,gid=999,mode=0700"
+        if hasattr(container, "with_tmpfs_mount"):
+            container.with_tmpfs_mount(data_dir, options)
+        else:
+            container.with_kwargs(tmpfs={data_dir: options})
 
     cmd = build_postgres_command(_config)
     if cmd is not None:

dinosaur/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "macrostrat.dinosaur"
-version = "4.2.0"
+version = "4.2.1"
 description = "Diff-based database migrations"
 authors = [{ name = "Daven Quinn", email = "dev@davenquinn.com" }]
 requires-python = ">=3.10,<4"

pyproject.toml

@@ -3,7 +3,7 @@ name = "macrostrat.python_libraries"
 version = "1.4.0"
 description = "Macrostrat Python libraries"
 authors = [{ name = "Daven Quinn", email = "dev@davenquinn.com" }]
-requires-python = ">=3.10,<3.14"
+requires-python = ">=3.11,<3.14"
 dependencies = [
     "setuptools>=82.0.1",
 ]
@@ -17,7 +17,11 @@ dev = [
     "macrostrat.package_tools",
     "macrostrat.utils",
     "macrostrat.auth_system",
+    "macrostrat.raster_index",
+    "macrostrat.raster_layers",
     "pytest>=7.2.2,<10",
+    "pillow>=10.0",
+    "mapbox-vector-tile>=2.1.0,<3",
     "python-dotenv>=1.0.0,<2",
     "requests>=2.27.1,<3",
     "rich>=13,<16",
@@ -40,6 +44,8 @@ default-groups = "all"
 "macrostrat.package_tools" = { path = "./package-tools", editable = true }
 "macrostrat.utils" = { path = "./utils", editable = true }
 "macrostrat.auth_system" = { path = "./auth-system", editable = true }
+"macrostrat.raster_index" = { path = "./raster-index", editable = true }
+"macrostrat.raster_layers" = { path = "./raster-layers", editable = true }
 
 
 [tool.pytest.ini_options]

raster-index/CHANGELOG.md

@@ -0,0 +1,6 @@
+# Changelog
+
+## [0.1.0] - 2026-08-12
+
+- Initial release: the `raster_layers` schema, `RasterIndex`, footprint
+  extraction through rio-tiler, bucket scanning, and a mountable Typer CLI

raster-index/README.md

@@ -0,0 +1,76 @@
+# `macrostrat.raster_index`
+
+An index of cloud-optimized rasters (COGs), grouped into named **layers** that a
+tile server can serve as single mosaics.
+
+Nothing here stores pixels. A row in `raster_layers.raster` is a reference to a
+COG in object storage plus the metadata needed to decide whether reading it is
+worthwhile for a given tile: its footprint, its native zoom range, and its data
+type. The schema name is `raster_layers` rather than `raster`/`rasters` to stay
+clear of PostGIS Raster's vocabulary.
+
+Serving these layers is [`macrostrat.raster_layers`](https://github.com/UW-Macrostrat/python-libraries/tree/main/raster-layers).
+
+## Usage
+
+```python
+from macrostrat.raster_index import RasterIndex, LayerDefinition
+
+index = RasterIndex("postgresql://localhost:5432/macrostrat")
+index.create_schema()  # or apply `schema_files()` through your own system
+
+index.register_layer(
+    LayerDefinition(slug="emit-minerals", name="EMIT mineral maps", maxzoom=14)
+)
+index.add_raster(
+    "https://storage.example.org/rasters/nevada.tif", layer="emit-minerals"
+)
+
+index.assets_for_tile(x=180, y=411, z=10, layers=["emit-minerals"])
+```
+
+## Schema
+
+`schema_files()` returns the SQL defining the schema, in application order, so a
+host application can fold it into its own schema management rather than calling
+`create_schema()`. Two tables and a few functions:
+
+- `raster_layers.layer` — a named mosaic, and the defaults its rasters inherit
+  (zoom range, rescale range, colormap).
+- `raster_layers.raster` — one COG: `href`, EPSG:4326 `footprint`, zoom range,
+  `dtype`/`nbands`/`nodata`, and the full reader metadata as `info`.
+- `raster_layers.get_rasters(x, y, z, layers[])` — asset selection, ordered by
+  layer priority then resolution. The core of the whole package.
+- `raster_layers.should_generate_tile(...)` — whether any asset actually resolves
+  at this zoom, for cache warmers and render short-circuits.
+- `raster_layers.layer_footprints(layers[])` — footprints as GeoJSON features.
+
+## CLI
+
+`raster-index` reads `RASTER_INDEX_DATABASE` (or `DATABASE_URL`):
+
+```sh
+raster-index define-layer emit-minerals --name "EMIT mineral maps"
+raster-index scan https://storage.example.org/remote-sensing-data/emit-mineral-maps/ \
+  --layer emit-minerals
+raster-index set-colormap emit-minerals --from https://storage.example.org/.../nevada.tif
+raster-index assets 10 180 411 --layer emit-minerals
+```
+
+The connection is a parameter of the app itself: `--database`, or those
+environment variables. A host application that already knows its own connection
+(Macrostrat mounts these as `macrostrat raster`) calls
+`set_default_connection(url_or_callable)` once, and its users never pass
+`--database` — though it still works, and still wins. Scanning object stores
+needs the `s3` extra (boto3), whichever URL form you use — an `https://` bucket
+URL is *rewritten* into an endpoint/bucket/prefix and listed through the same
+S3 API, rather than being a second code path.
+
+## Known limitations
+
+- Footprints are bounding boxes, so rasters crossing the antimeridian are
+  indexed incorrectly. The column is typed `geometry`, not `polygon`, so a
+  mask-derived footprint can replace them without a migration.
+- WebMercatorQuad only. Alternate tile grids (and non-Earth bodies, as in
+  [mars-tiler](https://github.com/davenquinn/mars-tiler)) would need a per-grid
+  bounds table.

raster-index/macrostrat/raster_index/__init__.py

@@ -0,0 +1,26 @@
+"""An index of cloud-optimized rasters, for mosaicked tile serving.
+
+Rasters live in object storage; this package records *where* they are, *what*
+they cover, and *which named layer* they belong to, so a tile server can answer
+"which files do I read for this tile?" with a single spatial query.
+
+Serving is a separate concern, handled by `macrostrat.raster_layers`.
+"""
+
+from .defs import LayerDefinition, RasterAsset, RasterInfo
+from .footprints import get_raster_info
+from .index import RasterIndex, schema_files
+from .scan import BucketPrefix, RasterObject, parse_bucket_url, scan_prefix
+
+__all__ = [
+    "RasterIndex",
+    "schema_files",
+    "get_raster_info",
+    "scan_prefix",
+    "RasterObject",
+    "BucketPrefix",
+    "parse_bucket_url",
+    "RasterAsset",
+    "RasterInfo",
+    "LayerDefinition",
+]