Introduce `unnormalized_reflectance` calibration level by mraspaud · Pull Request #3292 · pytroll/satpy

mraspaud · 2025-11-11T13:27:16Z

This PR introduce the unnormalized_reflectance calibration level to refer to reflectance not yet normalised by the solar pathlength, along with the corresponding standard name.

Closes Definition and accuracy of reflectance #52
Tests added
Fully documented
abi_l1b.yaml
abi_l1b_scmi.yaml
abi_l2_nc.yaml
acspo.yaml - no changes
agri_fy4a_l1.yaml
agri_fy4b_l1.yaml
ahi_hrit.yaml
ahi_hsd.yaml
ahi_l1b_gridded_bin.yaml - reference says the data are reflectances, not touched
ahi_l2_nc.yaml - no changes
ami_l1b.yaml
amsr2_l1b.yaml - no changes
amsr2_l2.yaml - no changes
amsr2_l2_gaasp.yaml - no changes
amsub_l1c_aapp.yaml - no changes
ascat_l2_soilmoisture_bufr.yaml - no changes
atms_l1b_nc.yaml - no changes
atms_sdr_hdf5.yaml - no changes
avhrr_l0_hrpt.yaml
avhrr_l1b_aapp.yaml
avhrr_l1b_eps.yaml
avhrr_l1b_gaclac.yaml
avhrr_l1c_eum_gac_fdr_nc.yaml - the channels are named reflectance_channel_X
aws1_mwr_l1b_nc.yaml - no changes
aws1_mwr_l1c_nc.yaml - no changes
caliop_l2_cloud.yaml - no changes
camel_l3_nc.yaml - no changes
clavrx.yaml - no changes
cmsaf-claas2_l2_nc.yaml - no changes
electrol_hrit.yaml - no changes
epic_l1b_h5.yaml - no changes
eps_sterna_mwr_l1b_nc.yaml
etm_l1_tif.yaml - "real" reflectance: Top of the Atmosphere Reflectance, which is obtained using the scene center sun angle
etm_l2_tif.yaml - surface reflectance
fci_l1c_nc.yaml
fci_l2_bufr.yaml - no changes
fci_l2_grib.yaml - no changes
fci_l2_nc.yaml - should this be left as-is even if the values are not the true reflectance? Johan and Martin decided to leave as is and report to EUM helpdesk the discrepancy
fy3a_mersi1_l1b.yaml
fy3b_mersi1_l1b.yaml
fy3c_mersi1_l1b.yaml
generic_image.yaml - no changes
geocat.yaml
gerb_l2_hr_h5.yaml - no changes
ghi_l1.yaml
ghrsst_l2.yaml - no changes
gld360_ualf2.yaml - no changes
glm_l2.yaml - no changes
gms5-vissr_l1b.yaml
goci2_l2_nc.yaml - assume actual reflectances
goes-imager_hrit.yaml
goes-imager_nc.yaml
gpm_imerg.yaml - no changes
grib.yaml - no changes
hsaf_grib.yaml - no changes
hsaf_h5.yaml - no changes
hy2_scat_l2b_h5.yaml - no changes
iasi_l2.yaml - no changes
iasi_l2_cdr_nc.yaml - no changes
iasi_l2_so2_bufr.yaml - no changes
iasi_ng_l2_nc.yaml - no changes
ici_l1b_nc.yaml - no changes
insat3d_img_l1b_h5.yaml
jami_hrit.yaml
li_l2_nc.yaml - no changes
maia.yaml - no changes
mcd12q1.yaml - no changes
meris_nc_sen3.yaml - no changes, real reflectances
mersi2_l1b.yaml
mersi3_l1b.yaml
mersi_ll_l1b.yaml
mersi_rm_l1b.yaml
mhs_l1c_aapp.yaml - no changes
mimicTPW2_comp.yaml - no changes
mirs.yaml - no changes
modis_l1b.yaml
modis_l2.yaml - no changes
modis_l3.yaml - no changes
msi_l1c_earthcare.yaml
msi_safe.yaml - no changes, L1c data are "real" reflectances
msi_safe_l2a.yaml - no changes, L2a data are "real" reflectances
mss_l1_tif.yaml
msu_gsa_l1b.yaml
mtsat2-imager_hrit.yaml
multiple_sensors_isccpng_l1g_nc.yaml
mviri_l1b_fiduceo_nc.yaml
mwi_l1b_nc.yaml - no changes
mws_l1b_nc.yaml - no changes
nucaps.yaml - no changes
nwcsaf-geo.yaml - no changes
nwcsaf-msg2013-hdf5.yaml - no changes
nwcsaf-pps_nc.yaml - no changes
oceancolorcci_l3_nc.yaml
oci_l2_bgc.yaml - no changes
olci_l1b.yaml
olci_l2.yaml
oli_tirs_l1_tif.yaml
oli_tirs_l2_tif.yaml
omps_edr.yaml
osisaf_nc.yaml - no changes
pace_oci_l1b_nc.yaml - no changes
safe_sar_l2_ocn.yaml - no changes
sar-c_safe.yaml - no changes
satpy_cf_nc.yaml
scatsat1_l2b.yaml - no changes
seadas_l2.yaml - no changes
seviri_l1b_hrit.yaml
seviri_l1b_icare.yaml
seviri_l1b_native.yaml
seviri_l1b_nc.yaml
seviri_l2_bufr.yaml - no changes
seviri_l2_grib.yaml - no changes
sgli_l1b.yaml
slstr_l1b.yaml
smos_l2_wind.yaml - no changes
tm_l1_tif.yaml - no changes
tm_l2_tif.yaml - no changes
tropomi_l2.yaml - no changes
vii_l1b_nc.yaml - this will be addressed in a future PR
vii_l2_nc.yaml - this will be addressed in a future PR
viirs_compact.yaml
viirs_edr.yaml - no changes
viirs_edr_active_fires.yaml - no changes
viirs_edr_flood.yaml - no changes
viirs_l1b.yaml
viirs_l2.yaml - no changes
viirs_sdr.yaml
viirs_vgac_l1c_nc.yaml
virr_l1b.yaml

codecov · 2025-11-11T14:29:49Z

Codecov Report

❌ Patch coverage is 99.20213% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.30%. Comparing base (18d7f60) to head (acbf68c).
⚠️ Report is 7 commits behind head on main.

Files with missing lines	Patch %	Lines
satpy/readers/viirs_compact.py	50.00%	2 Missing ⚠️
satpy/readers/seviri_l1b_hrit.py	75.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main    #3292    +/-   ##
========================================
  Coverage   96.30%   96.30%            
========================================
  Files         466      466            
  Lines       59090    59340   +250     
========================================
+ Hits        56908    57149   +241     
- Misses       2182     2191     +9

Flag	Coverage Δ
behaviourtests	`3.57% <0.00%> (-0.02%)`	⬇️
unittests	`96.39% <99.20%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

coveralls · 2025-11-12T14:26:12Z

Pull Request Test Coverage Report for Build 19459512598

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

0 of 0 changed or added relevant lines in 0 files are covered.
332 unchanged lines in 27 files lost coverage.
Overall coverage increased (+0.01%) to 96.392%

Files with Coverage Reduction	New Missed Lines	%
readers/abi_l1b.py	1	98.9%
readers/hrit_jma.py	1	99.33%
readers/hrpt.py	1	98.67%
writers/init.py	1	76.92%
readers/mersi_l1b.py	2	98.57%
readers/slstr_l1b.py	2	97.5%
tests/compositor_tests/test_aux_data.py	2	97.33%
tests/reader_tests/test_abi_l1b.py	2	98.97%
writers/geotiff.py	2	91.78%
readers/ami_l1b.py	3	97.39%

Totals
Change from base Build 19264575612:	0.01%
Covered Lines:	56802
Relevant Lines:	58928

💛 - Coveralls

sfinkens

Monumental effort 💪 The scissors idea really works well. Except for the calibration update, the YAML changes are mostly auto-formatting, right?

sfinkens · 2026-05-21T10:26:45Z

+        if dataset_id.get("calibration") == "reflectance":
+            warnings.warn("Reflectance is not a correct calibration for GMS-5/VISSR, "
+                          "please use 'unnormalized_reflectance'",
+                          DeprecationWarning)


This appears in many files, maybe factorize?

We are going to remove these just after release 0.61.0, so in this case I think it's better to have it explicitly in each affected reader.

sfinkens · 2026-05-21T10:27:33Z

+Satpy reads reflective data and carries two calibration levels for it: unnormalized_reflectance and reflectance.
+The difference is that reflectance is normalised by the cosine of the sun zenith angle, while the
+unnormalized_reflectance is not.
+
+Unnormalized_reflectance:


Suggested change

Satpy reads reflective data and carries two calibration levels for it: unnormalized_reflectance and reflectance.

The difference is that reflectance is normalised by the cosine of the sun zenith angle, while the

unnormalized_reflectance is not.

Unnormalized_reflectance:

Satpy reads reflective data and carries two calibration levels for it: ``unnormalized_reflectance`` and ``reflectance``.

The difference is that ``reflectance`` is normalised by the cosine of the sun zenith angle, while the

``unnormalized_reflectance`` is not.

Unnormalized reflectance:

pnuu

I didn't go through all the changes, so I only added one suggestion to FAQ to highlight the "programmatic" words.

Are all the pre-v1.0 warnings marked with the "scissors"?

What I saw (and did in the early stages my self) looks good in general.

gerritholl

Looks good, thanks for the work! Some small comments/questions inline.

gerritholl · 2026-05-21T10:37:29Z


 1. The geolocation is in a separate file, used for all bands
-2. VIS data is calibrated to Albedo (or reflectance)
+2. VIS data is calibrated to Albedo (or unnormalized_reflectance)


albedo is not reflectance or unnormalized reflectance, is there actually an albedo product (level 2)?

not that I know of, but I do not have experience with this reader.

gerritholl · 2026-05-21T10:41:55Z

 from satpy.readers.core.file_handlers import BaseFileHandler
 from satpy.utils import get_legacy_chunk_size

+# >8 v1.0


Is this comment in the wrong place?

indeed it is.

gerritholl · 2026-05-21T10:43:06Z

            raise ValueError("Calibration parameter should be radiance, "
-                             "reflectance or brightness_temperature")
+                             # 8< v1.0
+                             "reflectance, "


Maybe we shouldn't suggest a calibration parameter that is deprecated?

it's for backwards compatibility until we move on to 1.0

Yes, reflectance is still supported, but here we are suggesting to the user what calibrations to use, which is essentially documentation. I would argue that reflectance should work, but we should not mention that it works. A user might read this error message, decide to use reflectance, and then gets a warning that this is deprecated.

gerritholl · 2026-05-21T10:46:34Z

-
-        res.load(["VIS", "VNIR", "TIR1", "TIR3", "solar_azimuth_angle", "land_water_mask"])
+        #  8< v1.0
+        assert len(available_datasets) == 31


Why has the number of available datasets increased?

I guess because we added unnormalised reflectance items to the list

mraspaud · 2026-05-21T10:58:57Z

the YAML changes are mostly auto-formatting, right?

yes

Co-authored-by: Gerrit Holl <gerrit.holl@gmail.com>

Co-authored-by: Panu Lahtinen <pnuu+git@iki.fi>

mraspaud · 2026-05-21T11:04:11Z

I didn't go through all the changes, so I only added one suggestion to FAQ to highlight the "programmatic" words.

Are all the pre-v1.0 warnings marked with the "scissors"?

I hope so :)

gerritholl · 2026-05-21T11:06:05Z

One other thought: maybe the warning messages should include a message that the returned value for "reflectance" is not a reflectance due to the missing SZA normalisation, but is actually the unnormalised reflectance. The way it is now, users could get the impression that satpy so far returns reflectance but is switching to unnormalised reflectance (changed physics), whereas in reality we are only correcting the name.

ameraner

Great work, thank you! Just some notes on the FAQ.

ameraner · 2026-05-21T13:33:55Z

+
+Reflectance:
+
+ρ = L_vis × solar_irradiance / πcos(sza)


aren't we missing the Sun-Earth distance correction in these formulas?

Depends how you define the solar irradiance?

Yes, but the solar irradiance is pretty much always defined at 1AU, hence the need to have the distance correction term...

FYI the link to the FCI L1c guide with the formula: https://user.eumetsat.int/resources/user-guides/mtg-fci-level-1c-data-guide#ID-Converting-from-effective-radiance-to-reflectance-for-VNIR-channels

I think for clarity it's useful to include the Sun-Earth distance here

ameraner · 2026-05-21T13:35:24Z

+What are the reflectance vs unnormalized reflectance quantities for reflective channels
+---------------------------------------------------------------------------------------
+
+Satpy reads reflective data and carries two calibration levels for it: `unnormalized_reflectance` and `reflectance`.


a suggestion

Suggested change

Satpy reads reflective data and carries two calibration levels for it: `unnormalized_reflectance` and `reflectance`.

Satpy reads data from reflective/solar channels and carries two calibration levels for it: `unnormalized_reflectance` and `reflectance`.

…tance

Introduce calibration level

29b1cd2

mraspaud self-assigned this Nov 11, 2025

mraspaud added enhancement code enhancements, features, improvements refactor PCW Pytroll Contributors' Week labels Nov 11, 2025

mraspaud added 2 commits November 11, 2025 14:39

Make seviri hrit yaml more compact

e0af123

Fix hardcoded calibration enum values

8505da3

ameraner mentioned this pull request Nov 11, 2025

Introduce oblique_reflectance calibration #3272

Open

mraspaud added 5 commits November 12, 2025 10:31

Port fci reader to radiance_factor

83668d0

removesuffix doesn’t work

e43a62c

Reintroduce reflectance

d19d606

Add comment about stuff to remove

d44b8ea

Add scissors

1200fd4

mraspaud added 3 commits November 13, 2025 08:34

Reintroduce reflectance for seviri hrit (temporarily)

489b941

Fix seviri calibration tests to support radiance_factor

3d537b7

Add radiance_factor to native seviri

985a18d

mraspaud added this to Remote PCW Autumn 2025 and SatPy 1.0 Release Nov 13, 2025

mraspaud and others added 10 commits November 13, 2025 10:13

Add radiance_factor to virr_l1b

92c8124

Add radiance_factor to abi_l1b_scmi

624aec8

Add radiance_factor to abi_l1b

82a47d4

Replace reflectance with radiance_factor in abi_l2_nc

346dd2e

Replace reflectance with radiance_factor in agri_l1

e2fab1c

Replace reflectance with radiance_factor in ahi_hrit

f3dd90c

Replace reflectance with radiance_factor in ahi_hsd

683de53

Add radiance_factor to ami_l1b

5ac9a26

Adapt viirs sdr

de1c5d7

Add radiance_factor to avhrr_l0

d04e7cd

mraspaud added 3 commits May 20, 2026 13:06

Merge branch 'main' into unnormalized_reflectance

27ca267

Migrate isccpng

24ca446

Rename to unnrmlzd_rflctnce

0713543

mraspaud changed the title ~~Introduce radiance_factor calibration level and standard name~~ Introduce unnormalized_reflectance calibration level May 21, 2026

mraspaud marked this pull request as ready for review May 21, 2026 09:53

mraspaud requested review from adybbroe, ameraner, djhoese, gerritholl, pnuu, sfinkens and sjoro as code owners May 21, 2026 09:53

Fix wrong indentation

b17a788

sfinkens reviewed May 21, 2026

View reviewed changes

pnuu reviewed May 21, 2026

View reviewed changes

Comment thread doc/source/faq.rst Outdated

gerritholl approved these changes May 21, 2026

View reviewed changes

mraspaud and others added 2 commits May 21, 2026 13:00

Update doc/source/faq.rst

34e2fce

Co-authored-by: Gerrit Holl <gerrit.holl@gmail.com>

Update doc/source/faq.rst

0f62a41

Co-authored-by: Panu Lahtinen <pnuu+git@iki.fi>

mraspaud added 3 commits May 21, 2026 14:26

Fix ascii art position

6deaa75

Fix underline length

acbf68c

Merge branch 'main' into unnormalized_reflectance

37169d4

ameraner reviewed May 21, 2026

View reviewed changes

Change warning message for unnormalized_reflectance

5830940

strandgren mentioned this pull request May 28, 2026

Refactor code for Sun zenith angle corrections and change to effective_solar_pathlength_corrected instead of sunz_corrected for built-in RGB recipes #3397

Open

3 tasks

djhoese mentioned this pull request Jun 2, 2026

Add unnormalized_reflectance to VIIRS SDR (viirs_sdr) reader #3411

Open

mraspaud added 2 commits August 4, 2026 14:10

Merge remote-tracking branch 'upstream/main' into unnormalized_reflec…

f880bb7

…tance

Fix formulas

50308dc

	Satpy reads reflective data and carries two calibration levels for it: `unnormalized_reflectance` and `reflectance`.
	Satpy reads data from reflective/solar channels and carries two calibration levels for it: `unnormalized_reflectance` and `reflectance`.

Uh oh!

Conversation

mraspaud commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

coveralls commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 19459512598

Warning: This coverage report may be inaccurate.

Details

💛 - Coveralls

Uh oh!

sfinkens left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pnuu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gerritholl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mraspaud commented May 21, 2026

Uh oh!

mraspaud commented May 21, 2026

Uh oh!

gerritholl commented May 21, 2026

Uh oh!

ameraner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ameraner May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

mraspaud commented Nov 11, 2025 •

edited

Loading

codecov Bot commented Nov 11, 2025 •

edited

Loading

coveralls commented Nov 12, 2025 •

edited

Loading

ameraner May 21, 2026 •

edited

Loading