Skip to content

Fix CPR NL() off-by-one at the 87 deg polar boundary#65

Merged
affitz merged 1 commit into
mainfrom
fix/cpr-nl-87deg-polar-boundary
Jun 24, 2026
Merged

Fix CPR NL() off-by-one at the 87 deg polar boundary#65
affitz merged 1 commit into
mainfrom
fix/cpr-nl-87deg-polar-boundary

Conversation

@willi-werner

Copy link
Copy Markdown
Collaborator

Fixes an off-by-one in L0Latitude.NL() at exactly |lat| = 87.0 deg.

Problem

The topmost transition latitude T_LAT[57] rounds to exactly 87.0 deg and binarySearch matched it inclusively, so NL() returned 2 at |lat| = 87.0 deg, where the DO-260B standard (and the NASA/cpr formally-verified reference) require NL = 1 (polar cap). This changes longitude decoding for positions landing exactly on the 87 deg lattice point.

Fix

Treat the top transition latitude as exclusive for the NL=2 zone:

if (abslat >= T_LAT[T_LAT.length - 1]) return 1;   // |lat| >= 87 deg -> polar cap

Only the topmost boundary changes (it is the only NL transition latitude landing exactly on an exact integer-lattice point); the other 57 already matched NASA at every lattice point.

Validation

  • New regression test cpr/L0LatitudeTest.java: NL==1 at +/-87/88.5/90 deg, NL==2 just below 87 deg, NL==59 at the equator.
  • Full suite: 164 tests pass, 0 failures.
  • Cross-checked against the NASA/cpr DO-260B reference: NL boundaries agree to ~2 cm; an exhaustive lattice check around all 58 NL boundaries now matches; a 1,000,000-position airborne+surface decode cross-check against NASA agrees to ~5 mm.
  • Independently validated with GPT-5.5 (Codex).

Commit: 08b5ec6

The topmost transition latitude (87°) is the only NL boundary that lands
exactly on an L0 lattice point. Arrays.binarySearch matched it inclusively,
so NL() returned 2 at exactly |lat| == 87°, whereas DO-260B and the NASA
cpr reference require NL == 1 for |lat| >= 87° (polar cap). Make the top
boundary exclusive for the NL=2 zone.

Validated by an exhaustive lattice check (406 points across all 58 NL
boundaries) confirming this was the only divergence from NASA, and a 1M
random airborne+surface position cross-check against the NASA fixed-point
reference. Adds L0LatitudeTest covering the boundary; full suite 164 tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@affitz affitz merged commit 697b93b into main Jun 24, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants