Update for Euler and Cleanup - #69
Open
mjaehn wants to merge 72 commits into
Open
Conversation
…rom cdsapi. namelist setup is very similar to dyamond, preprocessing of initial conditions is supplied, minimal output postprocessing are handled. bc files need to be gathered and correctly linked (see icon_runjob_parent_global.cfg)
|
- environment.yml moved to the repository root, update both call sites - fix the undefined howtorun label - add era5_ic to the API function list and machines/ to the code tree - reformat the era5_ic docstring so Sphinx parses its lists
- set_machine now recognises santis (santis*, nid*) and daint is gone - submit_basic_python and wait_for_previous get santis branches modelled on eiger, and the daint ones are removed - Euler job scripts source machines/euler/setup_env.sh instead of activating a conda environment that a pip setup does not have - drop --partition on Euler, matching the runjobs where it was already removed - run_chain.py: remove the daint-only gpu/mc constraint assertion
The unconditional clone aborted the script under set -e on any re-run where ext/icon had been cloned before.
The Jenkinsfile was unusable (missing sh quoting, a script deleted in e39a29a, stale miniforge/miniconda PATHs) and Jenkins is no longer used. Rename jenkins/scripts/jenkins.sh to testing/run_tests.sh, move the remaining scripts to testing/scripts/ and update their internal paths, README and docs.
Also move the remaining ERA5 workflows next to the other icon entries and restore the file's original opening.
lredgrid_phys was .TRUE. with radiation_grid_filename pointing at the dynamics grid, i.e. a reduced radiation grid identical to the grid it reduces. Drop the redundant filename and switch the flag off.
- setup_env_euler.sh: orphaned duplicate of machines/euler/setup_env.sh - parent case tables/mypartab: byte-identical to partab_era5_to_icon.txt - machines/santis/modules.sh: only set variables setup_env.sh redefines Document that modules.sh is optional when adding a new machine.
era5_ic had a copy of the naming rules in jobs/icon.py; if the two ever drift, era5_ic writes a file ICON does not read. Move the rules into prepare_icon.get_inidata_filename() and call it from both jobs.
The spack build of the public ICON release does not necessarily produce the modules.env the runjobs source. Fall back to the machine modules with a warning, and have build_icon.sh report which of the two applies.
wget without -c left a truncated archive behind and created .tgz.1 on a retry, and a partial extraction only surfaced later as a missing input file. Resume the download, verify the files the test cases need, and stop run_tests.sh from aborting when input/ does not exist yet.
Piz Daint is no longer a supported system.
The namelist sets nproma itself; the shell variable was never read.
Rows extended with prepare_art_full_chem were never re-padded, so Sphinx errored on the table on every build. Predates the Euler work.
Use the same stack, compiler, MPI, cdo, nco and netcdf-c versions as the Euler site settings of cesm2icon, so the ICON executable we run is launched in the environment it was built against: https://github.com/C2SM/cesm2icon/blob/main/run/sites/euler.sh Python moves to 3.13.0 from the same stack. The era5_ic runjobs now source machines/euler/modules.sh instead of pinning their own cdo/nco versions.
Building ICON is no longer a step of the Processing Chain. Point the Euler cases at the persisting install /cluster/work/climate/icon_input/icon-model/release-2026.04-public for the executable, the radiation tables, ecrad data and the analysis varname map, and drop testing/scripts/build_icon.sh together with the build step in run_tests.sh. Because nothing produces ext/icon/modules.env any more, the runjobs source machines/euler/modules.sh directly, and icon-test-euler launches ICON with srun like the parent-global template and cesm2icon do. The input archive no longer has to carry the radiation tables and the varname map. The Kinne, CMIP6 volcanic and ozone boundary conditions are not part of the public release, so load_links.txt keeps them on a separate tree.
The 2023 input archive never contained a global R02B06 grid or ERA5 GRIBs,
and no ERA5 for 2018-01-01 exists on the cluster. Use what is there
instead:
- grid/extpar: icon_grid_0024_R02B06_G.nc and its matching extpar from the
DWD tutorial data (verified same uuidOfHGrid, grid_root=2, grid_level=6,
327680 cells). The previous 0002 name did not correspond to any grid.
- ERA5: era5_{ml,surf}_2013-05-25.grib, so the case period moves to
2013-05-25, the same period the parent-global case uses.
Both are copied into a shared directory by the new
testing/scripts/stage_icon-test-euler_input.sh and referenced by absolute
path, so the case needs nothing from the input archive. Drop the unused
meteo block, which pointed at data this workflow never reads.
A BASIC_PYTHON_JOB runs run_chain.py -s in a wrapper job, which submits the real job with 'sbatch --wait' and blocks. The wrapper's walltime therefore has to cover the queue time of the job it waits for. With walltime.prepare_icon at 10 minutes the wrapper was killed while its copy_input job was still queued, and era5_ic was worse: one key sized both the wrapper and the job it waits for, so it could never be satisfied. Add an optional walltime_jobs dictionary for the submitted jobs and leave walltime for the wrappers, raising the latter in the two Euler cases. Cases without walltime_jobs keep their previous behaviour.
Replace slang and casual phrasing (e.g. 'sanity check', 'do NOT', first-person 'we run', 'straight from') with more formal, precise wording. No functional changes.
setup_env.sh on Euler and Santis expected the venv at <repo_root>/venv by default, but nothing created it there, so activation silently failed and jobs crashed on missing dependencies. Default both scripts and all docs to <repo_root>/.venv, and remove the conda alternative (environment.yml, run_tests.sh --pip branch) to keep a single supported path.
inwp_radiation=1 (RRTM) is rejected as deprecated by the provided ICON binary, aborting the run at namelist crosscheck; switch to inwp_radiation=4 (ecRad), already used successfully in the parent-global-r2b7-era5-update-euler case. The run_icon.job script also let a crashed srun step exit silently: the trailing 'set +x' ran after it and its own success (exit 0) overwrote the exit status, so Slurm reported COMPLETED even though ICON had aborted. Capture srun's exit code and exit with it in both affected templates.
ICON derives the restart filename verbatim from master_model_nml's model_name, which every case sets to "ATMO". The hardcoded lowercase restart_atm_DOM01.nc never matched the actual restart_ATMO_DOM01.nc written by ICON, breaking restart lookup for any multi-chunk run.
The dynamics grid's companion -grfinfo.nc file is required for parallel patch decomposition but was never staged or referenced. ecRad also needs its own data directory, distinct from the legacy lrtm_filename, configured via a radiation_nml block.
output_nml previously had no variable list, so ICON accepted the output stream config but wrote zero variables (only the restart file was produced). Verified against a rerun: adds surface diagnostics and native 3D prognostic fields, matching the variables used in the parent-global-r2b7-era5-update-euler case.
era5_dir, the ICON grid files, and the ecRad Kinne/volcanic/ozone boundary conditions previously pointed into lroither's personal work directory. Switch them to the copies staged under icon_testing_input so the case is runnable without another user's tree being available.
The copy_input job script ran without 'set -e', so a failed rsync (e.g. a missing or unreadable source path) was silently ignored and the chain proceeded with incomplete input.
On eiger, 'constraint' takes gpu/mc, but on euler and santis it is a literal Slurm --constraint node-feature value (e.g. EPYC_7H12).
The era5_ic preprocessing job was hitting the 40-minute walltime and getting cancelled mid-run before completing.
Leftover intermediate files from a previous failed attempt caused ncks to prompt interactively for overwrite confirmation, which fails non-interactively in a batch job.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch adds Euler support to the processing chain, standardizes on a pip
.venv(conda dropped), and cleans up legacy build/config paths.icon-test-eulernow runs end-to-end on Euler with the provided ICONrelease-2026.04-publicbinary.Environment
.venvat repo root on both Euler and Santis; conda option andenvironment.ymlremoved$PROJECT-relative pathsicon-test-eulerfixes (chain previously failed or silently reported success on a crashed run)srunexit code was discarded, so a crashed ICON run showed up asCOMPLETED; the exit code is now propagated correctlyinwp_radiationfrom deprecated RRTM to ecRad, with correctecrad_data_path/radiation_nmlsetup-grfinfo.nccompanion grid file required for domain decompositionrestart_atm_...vs. ICON's actualrestart_ATMO_...) that broke chunk-to-chunk restartsml_varlistso the run actually writes atmospheric output, not just a restart fileBoth chunks (
2013052500_2013052506,2013052506_2013052512) now complete successfully with correct restart chaining.