Repository navigation
Conversation
Wires RasterDataSources.jl's new authenticated ARCO-ERA5 Zarr sources into the existing weather-loading framework, the same way ERA5 (the public GCP-hosted store) already works. ECMWFERA5 mirrors ERA5's binding directly (single base store). ECMWFERA5 Land's variables split across several per-topic Zarr stores instead of one shared store, so this adds a MultiGroupContiguousTimeSeries loader (native_group resolves each field to its group) rather than forcing it through the single-store ContiguousTimeSeries path; it falls back to ECMWFERA5 for cloud cover, the one quantity Land has no group for. _contiguous_series_coords/_contiguous_series_open now dispatch on the whole cloud-source object instead of a bare URL string, so the existing public-GCP and new authenticated paths can have separate implementations (MicroclimateMapperZarrExt) sharing only the CF-epoch-parsing helper. Verified by loading the module and running the new tests interactively against locally dev-linked RasterDataSources.jl/Rasters.jl checkouts (19/19 assertions); the existing ERA5 test coverage is unchanged by the weather.jl dispatch refactor.
Live testing crashed on ECMWFERA5Land: the 0-360 longitude wraparound
and descending-latitude assumptions in this function only held for the
old GCP ARCO-ERA5 store, not ECMWF's own stores (ascending latitude,
-180..180 longitude, confirmed against real store metadata). Fixes:
- Route the longitude conversion through the existing
longitude_convention/_native_lon_crop trait (already used elsewhere
in this file) instead of a hardcoded mod(x, 360); declares
longitude_convention(::Type{<:ERA5}) = Longitude360() since removing
the hardcode would otherwise regress the working GCP path.
- Replace the direction-assuming findfirst/findlast pair for latitude
with findall, which works regardless of ascending/descending order.
Added a regression test covering both conventions directly (synthetic
coords, no network), since this slipped through the existing offline
unit tests -- they never called this function with real-shaped data.
open_zarr_array had no disk cache, so every chunk of every variable was a fresh uncached HTTP request each time -- far slower than the CDS group-level cache open_zarr_store already provides.
The fixed 2° buffer (tuned for NCEP's ~1.9° grid) pulls a ~40x40 cell block per point at ERA5-Land's ~9km spacing -- hundreds of MB for a single point's time series. New points_load_buffer trait lets finer sources use a buffer sized to their own grid.
…guousTimeSeries getraster(ECMWFERA5) (no layer) falls through to RasterDataSources.jl's generic getraster(T) = getraster(T, layers(T)), and layers(ECMWFERA5) is its topic groups (:sfc, :wav) -- a NamedTuple of both, not the single source ContiguousTimeSeries expects. All fields live in :sfc.
ECMWFERA5's own :sfc group reports "seconds since ..." (confirmed live), unlike ERA5-Land's "hours since ...". Hardcoding an hour step misread seconds-offsets as hours, computing a time range 3600x too narrow -- 2 timesteps loaded instead of 8784, which _truncate_to_common_ti then propagated to every other field in the merged stack.
buffers.native was sized from variables(source) alone, missing any fallback-contributed field not already in the hardcoded WindSpeed/ ActualVapourPressure/Pressure extras. SILO/CRUCL2's wind_speed fallback happened to overlap those extras and never exposed this; ECMWFERA5Land's cloud_cover fallback does not, causing a FieldError on read. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…rces Two bugs, same class as the earlier points_load_buffer fix: (1) the init snapshot's fast path only special-cased ContiguousTimeSeries, so ECMWFERA5Land's stl1/swvl1 snapshot fell into the generic branch and requested a full year instead of ~1 hour; (2) vector.jl's init-soil and soil-moisture-source loads still used the hardcoded _POINTS_LOAD_BUFFER instead of the points_load_buffer trait. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds ERA5ECMWF and ERA5ECMWFLand as data sources. This requires a CDS token but then allows rapid extraction of time series for ERA5-based simulations, e.g. around 1.5 mins for complete download and run of a 1 year simulation.