Issue in soil map: isoil_data = 3 (HWSD mapping) provides wrong soil type maps.

[Ivanderkelen]

We discovered a major issue in the soil type map generated with HWSD base data to TERRA soil type mapping.

@StefanHagemann has created a fix for this using the updated HWSD 2.0 dataset. Scripts are provided in the extpar-data GIT for those with access: https://gitlab.dkrz.de/extpar-data/extpar-input-data/-/tree/master/raw_data_tools/soil/hwsd2?ref_type=heads

A Generated File with TERRA soil types on HWSD2 resolution is available at:
https://gitlab.dkrz.de/extpar-data/extpar-input-data/-/blob/master/soil/hwsd2_texture_terra.nc.xz?ref_type=heads

As this is not part of extpar yet, we suggest to remove the option to use isoil_data=3 to prevent other users from using wrong soil maps.

[Ivanderkelen]

Example of erroneous soil map created with isoil_data=3 for Belgium.

[stelliom]

Hi @Ivanderkelen,

Thank you for reaching out. Could you already localize the issue to a bug in the EXTPAR code or is it a problem with the raw input data? I am asking this because also isoil_type=2 uses the same data, so I would need to deactivate that too.

As far as I understood, using the dataset HWSD 2.0 solves/bypasses the issue. What does it entail to add this to EXTPAR: is it simply a matter of using this dataset instead of the old one or are there more changes required in the code?

Cheers,
Mikael

[StefanHagemann]

Hi Mikael,
I guess simply replacing the HWSD2 versions would not solve the problem as I did it differently than in EXTPAR. However, I provided my scripts to derive TERRA soil types from HWSD2.0 in the EXTPAR-data on DKRZ-Gitlab. The optimum solution would be to implement those scripts into EXTPAR. However, I never worked with EXTPAR so this is outside my capabilities. In the UDAG project it was said that at some point someone from DWD can do it. However, it seems that nobody was appointed to that task.
If you would be able to do that, it would be actually a fantastic solution.
Best regards
Stefan

[StefanHagemann]

forgot the @stelliom and @Ivanderkelen

[Ivanderkelen]

Hi @stelliom , fully agree with @StefanHagemann: I think the problem lies in both the EXTPAR code and the input data.

On the isoil_type=2 option, this does not produce the soil types that can directly be used in COSMO/ICON, as this option produces the soil types with the direct HWSD codes. So I would also remove this option at this point.

[stelliom]

Hi @StefanHagemann and @Ivanderkelen,

I currently don't have much time to work on this myself, however CSCS systems are down until Friday (18.09.26), so I might have 2-3 days to look into this. To do this however I would need more details, because I still don't understand what the problem is, or better put, what the cause is or if it's known.

1. Was there already an investigation to understand why the soil map created with isoil_type=3 is wrong? If yes, what was the outcome?
2. Did you already identify a specific line or algorithm in the EXTPAR code that has a bug?
3. Is the issue with the HWSD data on the extpar-input-data repo a matter of how it was downloaded/processed before storing it there or a problem with the raw dataset that cannot be solved (apart by using a newer version)?
4. How can I reproduce the results shown in the image above? Did you use Zonda or EXTPAR directly? Which version? Which namelist options?
5. @StefanHagemann Are these the instructions to do it your way? How do you use the resulting data in EXTPAR? Is it an input data for EXTPAR or does it replace what would be the EXTPAR output? The extpar-input-data repository should only contain input data for EXTPAR.

[StefanHagemann]

Dear @stelliom (cc to @Ivanderkelen),
in the UDAG project we found that the soil types based on HWSD1.1 from EXTPAR are unrealistic, especially over Scandinavia. A lot of problems were solved when using HWSD 2.0. Consequently I wrote a python routine that converts HWSD 2.0 into TERRA soil types with a subsequent interpolation to the ICON-CLM grid. Beate wrote a correction that takes into account different leake areas of Terra and HWSD2.0. This was done completely independent of EXTPAR. This was used in the UDAG project and assoicated ICON-CLM simulations of the community. Then I was asked to implement those routines in the extpar-data GIT at DKRZ. The connection of these data/routine with the extpar procedures was said (as mentioned above) that it will be done by DWD at some point. There was a strong desire at DWD to have HWSD2 based soil types in EXTPAR. As mentioned before I never used extpar myself. The intructions are written to inform userswhat the script is doing and how they can derive a soil type dataset for TERRA directly, which is consistent with extpar derived land sea mask. However, I naively guess if someone knows how extpar works, it shouldn' t be to difficult to implement the procedure.
Best regards
Stefan

[stelliom]

Ok, I see. Then the HWSD 2.0 data on extpar-input-data is not actually input data for EXTPAR, but it's bypassing it. That is not the place where this data should live, that repo should only contain raw input data for EXTPAR, which is responsible to process it (if needed) and remap it to the desired grid, not only a specific one. Sorry, I think I misunderstood what the data was at the time I merged the MR. Anyway, I think for now we can keep the data there until we come to the bottom of this, but the goal should be to implement this feature in EXTPAR and remove the data and scripts from the input data repo.

I am only maintaining EXTPAR (since last year) and integrating it with Zonda, but I am not responsible for its development (no resources at C2SM), therefore I also don't know exactly how all the components work, since these were developed by the community before I started working on it. Generally, if a user needs a new feature in EXTPAR they have to implement it themselves. However, since this seems to be tight to some sort of bug I will look into it in these days and hopefully it does not turn out to be very hard to implement, but I will probably have questions about what you need. However, before doing that I would also like to understand what the issue is with HWSD 1.1, since this is also used by other organizations and I would like to know if the issue is solvable. Therefore, if you have some answers to the questions in my previous message that would be very helpful.

we found that the soil types based on HWSD1.1 from EXTPAR are unrealistic, especially over Scandinavia

Do you think this is just due to the accuracy of the raw HWSD1.1 data or is it more likely a bug in the EXTPAR processing of the raw data?

[stelliom]

@StefanHagemann

[stelliom]

Example of erroneous soil map created with isoil_data=3 for Belgium.

@Ivanderkelen Would there be an image with a better resolution? I cannot read the labels.

[StefanHagemann]

Dear @stelliom , for your questions I refer to @Ivanderkelen . She did a very thorough analysis, and as far as I remember it is a mixture of both, the HWSD version and how TERRA soil types are generated in EXTPAR.

[Ivanderkelen]

@stelliom @StefanHagemann

I indeed did a very detailed dive into EXTPAR to find the issue. Most of your questions are answered by the document attached including maps, and references to the extpar code.

Summarized is the main reason of the bug located in the tex_coarse, tex_medium and tex_fine fields in the HWSD v1.1 raw input file. I could not trace back the source of these files from the raw HWSD v1.1 file.

On top of that is the way that TERRA soil types are derived (using zmix) questionable, and the fix of Stefan provides an alternative way of doing the classification. (The fix of is not (yet) included in the report.)

We will discuss this further next week at the SOILVEG meeting at the CLM Assembly. But for now, I would remove the HWSD v1.1 options from extpar as they produce really wrong results

comment_exptar_incorrect_soil_classification_HWSD.pdf

[stelliom]

Hi @Ivanderkelen,

Thank you for the detailed report, I will also look into it and see if I can get to the source of the files. However, I cannot remove the HWSD v1.1 options completely just now, because these seem to be used for MeteoSwiss operational setups. For now we will add a warning on Zonda and I will find a way to make it apparent for people that use EXTPAR directly. Also, do you know which version of EXTPAR you used for your investigation? Just in case this was solved in a later version already and we are chasing a non-issue (I will anyway redo your tests myself with the latest version, so I guess I will discover that then).

[Ivanderkelen]

Hi @stelliom,

That sounds good. We used v5.15.1, but I am pretty confident the issue is not fixed in later versions (I did a search in the github issues).

[stelliom]

Ok, yes indeed I don't think this was fixed since 5.15.1. I just wanted to make sure you were not using a super old version.