Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
b609da6
Add Level 1 format option
Sep 17, 2025
44265f2
Add Level 2 format option
Oct 31, 2025
bc35eab
Update README and docs
Oct 31, 2025
f3b826c
Fixed reading in l1/spc files and time for Cloudnet format
Nov 12, 2025
3ac67e4
Fix test
Jul 1, 2026
393040c
Fix type in plotting function
Jul 1, 2026
3ba0769
Implement final E-Profile data format (Level 2)
Jul 21, 2026
1cc4cca
Fix array type
Jul 21, 2026
114a355
Account for different height variable
Jul 21, 2026
9bc402d
Provide instrument type with site name
Jul 21, 2026
2531d75
Add Level 1 format option
Sep 17, 2025
0772902
Add Level 2 format option
Oct 31, 2025
b8b793d
Update README and docs
Oct 31, 2025
7c95dfa
Fixed reading in l1/spc files and time for Cloudnet format
Nov 12, 2025
5e0c567
Fix test
Jul 1, 2026
dc4f487
Implement final E-Profile data format (Level 2)
Jul 21, 2026
7b2660f
Fix array type
Jul 21, 2026
c1727c4
Account for different height variable
Jul 21, 2026
63b0a89
Provide instrument type with site name
Jul 21, 2026
a403c6a
Merge remote-tracking branch 'origin/no_config' into no_config
Jul 30, 2026
d3c252f
Improve usage of different data formats and update docs
tobiasmarke Aug 25, 2026
98ea6f8
Fix general quality flag
tobiasmarke Aug 27, 2026
6a3b59e
Make site config optional for all data formats
tobiasmarke Aug 27, 2026
10ab697
Infer instrument type from lev1; data format specific metadata
tobiasmarke Sep 2, 2026
2d082f2
Update README and docs
tobiasmarke Sep 2, 2026
836b32c
Fix test
tobiasmarke Sep 2, 2026
8ff270c
Merge branch 'main' into no_config
Sep 9, 2026
01ca741
Fix height variable for interpolation
Sep 9, 2026
c2dcac2
Restore lev1_to_nc argument order and keep returned Rpg object intact
tukiains Sep 17, 2026
8c2d0f8
Add test for processing without site config
tukiains Sep 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 21 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ Level 2 data products and visualization and is based on the IDL code
[mwr_pro](https://zenodo.org/records/7973553).

The netCDF data format including metadata information, variable names and file naming
is designed to be compliant with the data structure and naming convention
developed in the [EUMETNET Profiling Programme E-PROFILE](https://www.eumetnet.eu/).
is designed to be compliant with either the data structure and naming convention
developed in the [EUMETNET Profiling Programme E-PROFILE](https://www.eumetnet.eu/), or within [ACTRIS Cloudnet](https://cloudnet.fmi.fi/).

MWRpy documentation: <https://actris-cloudnet.github.io/mwrpy/>

Expand Down Expand Up @@ -51,31 +51,34 @@ MWRpy requires Python 3.10 or newer.

## Configuration

The folder `mwrpy/site_config/` contains configuration files for each instrument
type, which defines the input and output data paths etc.
The folder `mwrpy/site_config/` contains mandatory configuration files for each instrument
type, which also define the input and output data paths, metadata, etc.
For example, this is the [configuration file for RPG-HATPRO](mwrpy/site_config/hatpro.yaml).

The folders for each site, e.g. `mwrpy/site_config/hyytiala/`, contain a
site and instrument specific configuration file (`config.yaml`) and retrieval coefficients.
For example, this is the [configuration file for Hyytiälä](mwrpy/site_config/hyytiala/config.yaml).
The folders for each site, e.g. `mwrpy/site_config/hyytiala/`, contain a folder with the required retrieval coefficients
(`mwrpy/site_config/hyytiala/coefficients/`) and an optional site and instrument specific configuration file (`config.
yaml`). For example, this is the [configuration file for Hyytiälä](mwrpy/site_config/hyytiala/config.yaml), which can
help with configuring multiple instruments of the same type.

## Command line usage

MWRpy can be run using the command line tool `mwrpy/cli.py`:

usage: mwrpy/cli.py [-h] -s SITE [-d YYYY-MM-DD] [--start YYYY-MM-DD]
[--stop YYYY-MM-DD] [-p ...] [{process,plot}]
[--stop YYYY-MM-DD] [-f ...] [-p ...] [{process,plot}]

Arguments:

| Short | Long | Default | Description |
| :---- | :----------- | :------------------ | :--------------------------------------------------------------------------------- |
| `-h` | `--help` | | Show help and exit. |
| `-s` | `--site` | | Site to process data from, e.g, `hyytiala`. Required. |
| `-d` | `--date` | | Single date to be processed. Alternatively, `--start` and `--stop` can be defined. |
| | `--start` | `current day - 1` | Starting date. |
| | `--stop` | `current day ` | Stopping date. |
| `-p` | `--products` | 1C01, single, multi | Processed products, e.g, `1C01, 2I02, 2P03, single`, see below. |
| Short | Long | Default | Description |
| :---- | :------------- | :------------------------ | :--------------------------------------------------------------------------------- |
| `-h` | `--help` | | Show help and exit. |
| `-s` | `--site` | | Site to process data from, e.g, `hyytiala`. Required. |
| `-d` | `--date` | | Single date to be processed. Alternatively, `--start` and `--stop` can be defined. |
| | `--start` | `current day - 1` | Starting date. |
| | `--stop` | `current day ` | Stopping date. |
| `-p` | `--products` | `1C01`, `single`, `multi` | Processed products, e.g, `1C01, 2I02, 2P03, single`, see below. |
| `-f` | `--format` | `cloudnet` | Data format to be used (`cloudnet`, `e-profile`). |
| `-i` | `--instrument` | `hatpro` | Instrument type to be processed (`hatpro`, `lhatpro`, `lhumpro_u90`). |

Commands:

Expand Down Expand Up @@ -109,6 +112,8 @@ Commands:
- single: Single pointing data product (including 2I01, 2I02, 2I06, 2P01, 2P03, and derived products)
- multi: Multiple pointing data product (including 2P02, and derived products)

Only the `1C01`, `single`, and `multi` data types are available when using the Cloudnet file format.

## Licence

MIT
84 changes: 84 additions & 0 deletions docs/source/command_line_usage.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
==================
Command line usage
==================

With the instrument type configuration (``mwrpy/site_config/{instrument_type}.yaml``) and retrieval files in
``mwrpy/site_config/{site}/coefficients/``, MWRpy can also be run using the command line tool `mwrpy/cli.py`:

.. code-block::

mwrpy/cli.py [-h] -s SITE [-d YYYY-MM-DD] [--start YYYY-MM-DD]
[--stop YYYY-MM-DD] [-p ...] [{process,plot}]

.. list-table:: Arguments
:widths: 10 20 20 50
:header-rows: 1

* - Short
- Long
- Default
- Description
* - `-h`
- `--help`
-
- Show help and exit.
* - `-s`
- `--site`
-
- Site to process data from, e.g, `hyytiala`. Required.
* - `-d`
- `--date`
-
- Single date to be processed. Alternatively, `--start` and `--stop` can be defined.
* -
- `--start`
- `current day - 1`
- Starting date.
* -
- `--stop`
- `current day`
- Stopping date.
* - `-p`
- `--products`
- 1C01, single, multi
- Processed products, e.g, `1C01, 2I02, 2P03, single`, see Data Types.
* - `-f`
- `--format`
- cloudnet
- Data format to be used (`cloudnet`, `e-profile`).
* - `-i`
- `--instrument`
- hatpro
- Instrument type to be processed (`hatpro`, `lhatpro`, `lhumpro_u90`).

These commands are available to select the processing mode:

.. list-table:: Commands
:widths: 20 30
:header-rows: 1

* - Command
- Description
* - `process`
- Process data and generate plots (default).
* - `plot`
- Only generate plots.
* - `no-plot`
- Only generate products.
* - `reprocess`
- Like `process`, but skips days when data processing fails.

Example usage
-------------
To process and plot Level 1 & 2 data (1C01, single, multi) for the site `Hyytiala` (HATPRO instrument) for April 6,
2023, in the Cloudnet format, run:

.. code-block::

python mwrpy/cli.py -s hyytiala -d 2023-04-06

Run the following command for the E-Profile format and no plots:

.. code-block::

python mwrpy/cli.py -s hyytiala -d 2023-04-06 -f e-profile no-plot
31 changes: 31 additions & 0 deletions docs/source/data_types.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
==========
Data Types
==========

The following data types (E-PROFILE naming convention) are available in MWRpy for Level 1 and Level 2 products:

Level 1
.......

- 1B01: MWR brightness temperatures from .BRT and .BLB/.BLS files + retrieved spectrum
- 1B11: IR brightness temperatures from .IRT files
- 1B21: Weather station data from .MET files
- 1C01: Combined data type with time corresponding to 1B01

Level 2
.......

- 2I01: Liquid water path (LWP)
- 2I02: Integrated water vapor (IWV)
- 2I06: Stability Indices
- 2P01: Temperature profiles from single-pointing observations
- 2P02: Temperature profiles from multiple-pointing observations
- 2P03: Absolute humidity profiles
- 2P04: Relative humidity profiles (derived from 2P01/2P02 + 2P03)
- 2P07: Potential temperature (derived from 2P01/2P02 + 2P03)
- 2P08: Equivalent potential temperature (derived from 2P01/2P02 + 2P03)
- single: Single pointing data product (including 2I01, 2I02, 2I06, 2P01, 2P03, and derived products)
- multi: Multiple pointing data product (including 2P02, and derived products)


The data types 1C01, single, and multi are also available in the Cloudnet format.
120 changes: 110 additions & 10 deletions docs/source/fileformat.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,20 +6,28 @@ All MWRpy files use ``NETCDF4_CLASSIC`` data model, i.e., ``HDF5`` file format.
**Dimensions**

.. list-table::
:widths: 25
:widths: 25 25
:header-rows: 1

* - Name
* - Name (E-PROFILE)
- Name (Cloudnet)
* - time
- time
* - bnds
-
* - frequency
- frequency
* - ir_wavelength
- ir_channel
* - receiver_nb
- receiver_nb
* - t_amb_nb
- t_amb_nb
* - height
- height


**Variables (common to all files)**
**Variables (common to all E-PROFILE files)**

.. list-table::
:widths: 25 50 25 25 25 25
Expand All @@ -43,25 +51,62 @@ All MWRpy files use ``NETCDF4_CLASSIC`` data model, i.e., ``HDF5`` file format.
- seconds since 1970-01-01 00:00:00.000
- int32
-
* - latitude
* - station_latitude
- Latitude of measurement station
- time
- degree_north
- float32
- latitude
* - longitude
* - station_longitude
- Longitude of measurement station
- time
- degree_east
- float32
- longitude
* - altitude
* - station_altitude
- Altitude above mean sea level of measurement station
- time
- m
- float32
- altitude

**Variables (common to all Cloudnet files)**

.. list-table::
:widths: 25 50 25 25 25 25
:header-rows: 1

* - Name
- Long name
- Dimensions
- Units
- Data type
- Standard name
* - time
- Time UTC
- time
- hours since YYYY-MM-DD 00:00:00 +00:00
- double
- time
* - latitude
- Latitude of site
- time
- degree_north
- float32
- latitude
* - longitude
- Longitude of site
- time
- degree_east
- float32
- longitude
* - altitude
- Altitude of site
- time
- m
- float32
- altitude


MWR-Level 1 files
.................
Expand All @@ -70,7 +115,8 @@ MWR-Level 1 files
~~~~~~~~~

The Level 1 default file type ``1C01`` contains all variables from the file types
``1B01``, ``1B11`` (if an infrared radiometer is available), and ``1B21`` (if a weather station is available).
``1B01``, ``1B11`` (if an infrared radiometer is available), and ``1B21`` (if a weather station is available) and is
available for the E-PROFILE and Cloudnet data format.

**Variables (MWR_1B01 specific)**

Expand Down Expand Up @@ -193,6 +239,25 @@ The Level 1 default file type ``1C01`` contains all variables from the file type
- int32
-

**Additional Cloudnet variable**

.. list-table::
:widths: 25 50 25 25 25 25
:header-rows: 1

* - Name
- Long name
- Dimensions
- Units
- Data type
- Standard name
* - zenith_angle
- Zenith angle
- time
- degree
- float32
- zenith_angle

**Variables (MWR_1B11 specific)**

.. list-table::
Expand Down Expand Up @@ -242,6 +307,25 @@ The Level 1 default file type ``1C01`` contains all variables from the file type
- float32
-

**Additional Cloudnet variable**

.. list-table::
:widths: 25 50 25 25 25 25
:header-rows: 1

* - Name
- Long name
- Dimensions
- Units
- Data type
- Standard name
* - ir_zenith_angle
- Infrared sensor zenith angle
- time
- degree
- float32
-

**Variables (MWR_1B21 specific)**

.. list-table::
Expand Down Expand Up @@ -298,7 +382,7 @@ The Level 1 default file type ``1C01`` contains all variables from the file type
-

MWR-Level 2 files
...............
.................

**Variables (common to all Level 2 files)**

Expand All @@ -324,12 +408,26 @@ MWR-Level 2 files
- degree
- float32
- sensor_elevation_angle
* - quality_flag
- General quality flag
- time
- 1
- int32
-
* - quality_flag_status
- General quality flag status
- time
- 1
- int32
-

Single pointing file
~~~~~~~~~~~~~~~~~~~~

The Level 2 default file type ``single`` contains all variables from the file types
``2I01``, ``2I02``, ``2I06``, ``2P01``, and ``2P03`` (if the respective retrieval coefficients are available).
``2I01``, ``2I02``, ``2I06``, ``2P01``, and ``2P03`` (if the respective retrieval coefficients are available) and is
available for the E-PROFILE and Cloudnet data format. The variable / dimension ``height`` is named ``altitude`` in
the E-PROFILE data format.

**Variables (MWR_2I01 specific)**

Expand Down Expand Up @@ -545,7 +643,9 @@ Multiple pointing file
~~~~~~~~~~~~~~~~~~~~~~

The Level 2 default file type ``multi`` contains all variables from the file types
``2P02``, ``2P04``, ``2P07``, and ``2P08`` (if the respective retrieval coefficients are available).
``2P02``, ``2P04``, ``2P07``, and ``2P08`` (if the respective retrieval coefficients are available) and is
available for the E-PROFILE and Cloudnet data format. The variable / dimension ``height`` is named ``altitude`` in
the E-PROFILE data format.

**Variables (MWR_2P02 specific)**

Expand Down
Loading