You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A module-by-module tour of all three services. Paths are relative to the repo
root. Line counts are approximate (as of this writing) to signal where the
weight is. This file is a complete narrative map on its own; the generated
code-API reference adds browsable signatures on
top.
weather_api/ FastAPI service (the API + all data logic + CLIs)
key_manager_ui/ Streamlit admin UI
postgis_db/ PostGIS container image + DB provisioning
Service: weather_api/ (the API)
Source lives under weather_api/src/weather_api/. Entry point is main.py.
Top-level app & wiring
Module
Lines
Role
main.py
133
Builds the FastAPI app: mounts the data router and the /admin sub-app, sets up the perf middleware, /, /health, /favicon.ico, and the big /docs description text (including the async-job usage examples). Imports MAX_TIME_RANGE_DAYS from api.schema for the description (single source, default 7).
dependencies.py
7
Instantiates the three shared singletons imported everywhere: dask_client, postgis (PostGISRasterInterface), s3 (S3Interface). Importing this starts a local Dask cluster.
lifespan.py
64
Startup/shutdown: seeds the auth DB, then runs two loops — hourly cache refresh and daily rollover.
config.py
21
Loads api_config.json into a Pydantic Config (file paths/URLs for projection params, state shapefile, EPSG CSV).
get_new_or_existing_dask_client() — connect to an existing scheduler or spin up a LocalCluster.
The api package — HTTP layer
Module
Lines
Role
api/routes.py
178
The five endpoints. point/boundingbox/state share _handle_data_request, which calls count_missing_for_request → either returns 202 (create job + background task) or runs get_data and returns 200. Also jobs/{id} (status) and jobs/{id}/result (download). Auth is a router-wide Depends(verify_api_key).
api/engine.py
146
get_data() orchestrates a request: fetch (via hrrr.data.fetch_data), and for ALL fan out across variables in a thread pool; then format_result_da() serializes to JSON / NetCDF / GeoTIFF. (Time coordinates are normalized to tz-naive UTC upstream in postgis/interface.py so NetCDF/JSON serialize consistently.)
api/schema.py
278
The request models (PointRequest, BBoxRequest, StateRequest) and their validation: flexible datetime parsing, time-range + archive-bounds limits, and CONUS-bounds checks via EPSGBounds (reads the EPSG CSV). Also JobResponse, CONUSPoint/CONUSBoundingBox, and shape_to_str.
The hrrr package — the weather-data core
Module
Lines
Role
hrrr/schema.py
165
HRRRVar enum + metadata (units, level, long name, s3_root, calculated, dependent_vars) and the CONUSState enum (48 states). The single source of truth for "what variables exist."
hrrr/data.py
385
The fetch pipeline: async reads from HRRRZarr (async_conus_hrrr_zarr_var), the calculated-var path (async_calculate_from_cached_deps), the download→S3→PostGIS pipeline (upload_missing_data_to_s3_and_db, pipeline_single_timestamp), _should_store_indb, and the synchronous entry point fetch_data() used by the API and jobs.
hrrr/calculated.py
134
wind_speed/wind_direction and the calculate() dispatcher that builds WSPD/WDIR datasets from UGRD/VGRD. The quadrant/arctan2 math and the calm-wind→NaN rule live here.
Storage layer
Module
Lines
Role
s3.py
347
S3Interface (boto3) for the private bucket: upload a dataset as a GeoTIFF (LZW, predictor=3, 128² tiles), download, list. Plus key helpers: s3_object_name (YYYY/MM/DD/HH/VAR_YYYYMMDDHH.tif), full_s3_url (/vsis3/... for PostGIS), sanitize_data_for_serialization.
postgis/interface.py
559
PostGISRasterInterface — the raster catalog gateway. register_raster (shells out to raster2pgsql, in-DB vs out-DB), get_missing_timestamps, single_raster_data_exists, fetch (dispatches to the three SQL queries), rollover_to_outdb, run_maintenance (GIST index + ANALYZE), and the result→DataArray converters incl. postgis_output_data_filter (physical-plausibility NaN filtering). Also ConnectionString.
postgis/queries.py
87
The three parameterized SQL templates: point_query (ST_Value, one float per hour), bbox_query and state_query (ST_AsGDALRaster GeoTIFF bytes, clipped/unioned/reprojected).
Jobs & caching CLIs
Module
Lines
Role
jobs.py
366
The async batch-job system: count_missing_for_request (200 vs 202 + parallelism-adjusted work estimate), create_batch_job, estimate_job_seconds (history-based ETA), run_batch_job (semaphore, status transitions, cancellation), format_result_to_bytes. BatchJob rows live in the auth DB.
cache.py
196
cache_range / _cache_single_var — the pre-population engine used by the CLI and the hourly refresh. Sequential-per-variable, retry with exponential backoff, and an S3-present/PostGIS-absent "repair pass." Classifies each hour ok/not_in_archive.
cache_cli.py
154
python -m weather_api.cache_cli --start … --end … [--retention-months N] [--max-retries N] [--postgres-host …]. No --in-db flag (contrary to the old README). Prints a per-variable report; exits non-zero if any hour was not_in_archive.
restore.py
130
restore_from_s3 — rebuild the PostGIS catalog by scanning every GeoTIFF in the bucket, parsing VAR/timestamp from the key, and re-registering (respecting retention). parse_s3_key is the pure, easily-testable core.
restore_cli.py
115
python -m weather_api.restore_cli [--dry-run] [--force] [--retention-months N] [--workers N]. Also pixi run restore.
The auth package
Module
Lines
Role
auth/models.py
116
SQLAlchemy models for the auth DB: User, UserSession, APIKey, RegistrationToken, PasswordResetToken, APIKeyUsageRecord, BatchJob. See database-schema.md.
auth/database.py
108
Engine/session factory (authdb_engine, authdb_session, get_auth_db) and init_auth_db (creates the DB + tables, seeds the admin user and admin API key from env).
record_usage — writes an APIKeyUsageRecord as a background task.
auth/email.py
76
SMTP registration-confirmation and password-reset emails.
auth/routes_admin.py
1146
The admin API (see below).
auth/routes_admin.py (1146 lines) — the admin API
A separate FastAPI app (key_admin_app) mounted at /admin, called by the
Streamlit UI. docs_url=None, so its routes are not in /openapi.json.
Groups of endpoints:
Auth/session:POST /register, GET /confirm-registration,
POST /login, GET /logout, POST /reset-password,
POST /request-password-reset, GET /confirm-password-reset,
POST /complete-password-reset.
Dashboard:GET /dashboard.
Users (admin):GET /users (search/paginate), GET /users/{id}/keys,
POST /users/bulk-delete, POST /users/{id}/promote,
POST /users/{id}/delete.
Keys:POST /keys/generate (self-service, respects key_count_limit),
POST /keys/{id}/deactivate|reactivate|delete, GET /all-keys.
Usage (admin):GET /usage/summary, GET /usage/search.
Raster catalog (admin):GET /catalog-stats, GET /catalog-coverage
(these query the raster DB directly via _postgis_conn, using
POSTGRES_HOST/POSTGRES_PORT/CATALOG_QUERY_TIMEOUT_MS).
Batch jobs (admin):GET /batch-jobs/active, GET /batch-jobs/stats,
GET /batch-jobs/history, POST /batch-jobs/{job_id}/cancel.
Matplotlib visualizations of the perf log (per-request Gantt, step-duration box/KDE plots). Not on the request path.
Data & config files
weather_api/api_config.json — paths/URLs for HRRR projection params, the
state shapefile, and the EPSG CSV.
weather_api/data/conus_epsg_bounds_list.csv — ~2,065 EPSG codes with their
CONUS-intersection bounding boxes (pipe-delimited). Drives projection
validation.
weather_api/notebooks/*.ipynb — demo/illustrative only (not imported by
the app). wind.ipynb is the original WSPD/WDIR derivation prototype and is
worth reading for tacit context; the rest are scratch/experiments.
Service: key_manager_ui/
A Streamlit multi-page app (src/key_manager_ui/). See
../end-user/web-ui-guide.md for the functional
walkthrough.
File
Lines
Role
app.py
36
Entry point: page config, session-state init, st.navigation (admin pages shown only to admins).
api_client.py
—
Shared httpx client factory + safe_get (resilient GET: surfaces transport/5xx errors, clears the session only on 401/403).
pages/Home.py
173
Login/register/reset UI + the user Dashboard (view/generate own keys, change password).
pages/1_Admin_User_Management.py
293
User search, promote, delete; per-user key management.
pages/2_Admin_Key_Usage.py
346
Usage summary + record search; key activate/deactivate/delete.
pages/3_Admin_Raster_Catalog.py
241
Catalog stats + coverage report (charts).
pages/4_Admin_Batch_Jobs.py
566
Active jobs, cancel, history + charts.
Notes: reads API_URL (duplicated across the page files) and
MINIMUM_PASSWORD_LENGTH; runs on port 8080; declares
streamlit/httpx/pandas/altair/numpy. Pages share get_client/safe_get
from api_client.py.
Service: postgis_db/
The database image and its first-boot provisioning. Provisioning scripts run
only on an empty data directory (first init).
File
Role
Dockerfile
postgis/postgis:17-3.5 + curl/unzip for the shapefile download.
docker-entrypoint-initdb.d/01-provisioning.sql
Enables postgis + postgis_raster, enables out-DB rasters + all GDAL drivers, registers the custom SRID 990099 (HRRR LCC), creates conus_raster_catalog + its indexes.