#14510 Nested hybrid grid: prototype FIPNEST parent-child arrays thro…

[kriben]

…ugh the INIT file

Adds a compact FIPNEST/FIPSLOT/REFINE encoding of the nested hybrid grid parent-child relations that can travel inside the INIT file instead of the OLDIJK sidecar: FIPNEST is the 1-based flat natural index of each cell's immediate parent (refined-away host slots carry their own link so every chain terminates at a coarse cell), FIPSLOT packs the within-parent i/j/k offsets, and REFINE is the existing per-cell level. FIPNEST alone is provably insufficient: the refinement level is not the parent-chain depth (L2 and L3 both refine the coarse grid directly in the DROGON dataset), and within-parent placement is ambiguous from flat coordinates when a level has hole layers.

RigNestedHybridGridFipnestCodec derives the arrays from the sidecar input by mirroring the reconstructor's level classification, and synthesizes an equivalent reconstruction input back from them (identical up to per-level constant TMP shifts, which reconstruct() is invariant to), so reusing the reconstructor unchanged guarantees that a grid imported through FIPNEST equals the sidecar-based import. A successful sidecar reconstruction auto-exports _FIPNEST.grdecl next to the grid file, scripts/nested_hybrid_grid/ insert_fipnest_into_init.py inserts the arrays into an INIT file using resfo, and RimEclipseResultCase prefers the INIT-embedded arrays over the sidecars when both exist.

Verified by unit tests covering the codec round-trip, array invariants, cellwise grid equality of the two import paths, the GRDECL writer, and a hermetic end-to-end pass that stages the model, appends the INTE arrays to the INIT copy and imports without sidecars.

[kriben]

Some musings by claude:

❯ How much will I have to change to make this approach work if I get the correctly encoded indexed in FIPNEST and drop the FILSLOT and REFINE in the INIT?                                                                                                     

● Good question — the honest answer is: the plumbing barely changes, but the decoder half of the codec would need a rewrite, and one of the two dropped arrays loses information that cannot be recovered from FIPNEST at all. Let me break it down by what
  each array actually pays for.

  What survives unchanged

  The INIT probing, the RimEclipseResultCase precedence wiring, the Python inserter (--keywords FIPNEST already works), the reconstructor itself, and the chain-walk/kFactor logic in the decoder — all of that is FIPNEST-driven already. The change is
  concentrated in RigNestedHybridGridFipnestCodec::buildInputFromParentChildArrays() (roughly 200–300 lines of rework) plus relaxing the "all three keywords required" gate, and reworking the equality tests.

  Dropping REFINE — works only if your data is strictly nested

  Chain depth (walk FIPNEST until 0) equals the refinement level if and only if every level-n cell's parent is a level-(n-1) cell. From our earlier discussions, your real target data is a strict chain 5+ levels deep — for that data, REFINE is genuinely
  redundant and this is a small derivation loop.

  DROGON is the counterexample: L2 and L3 both point directly at coarse cells (depth 2 for both). Merging them into one "level" feeds mixed refinement factors into the uniformity check, the whole merged level gets deferred, and nothing reconstructs. To
  handle DROGON-style data without REFINE you'd need a clustering step instead of levels: group cells by parent, then cluster parent cells by per-parent refinement-factor signature plus coarse-space connectivity. That's the main new code (~100–150 lines),
  it's heuristic at the margins, and the LGR decomposition/naming can come out different from today's one-LGR-per-level even when the geometry is right. The derived levels would also need to be stored as the REFINE result so the per-level aggregation
  helpers keep working.

  Dropping FIPSLOT — placement info is genuinely gone, not just inconvenient

  FIPSLOT only matters for primary levels (the ones refining the coarse grid). Nested levels are already placed from flat-coordinate offsets, so for a strict deep chain only level 2 needs it. But where it's needed, it's irrecoverable: DROGON's L2 children
  sit at within-parent K slots {0,2} of factor 4 (hole layers), which is indistinguishable in flat coordinates from {0,1} of factor 2 — and geometry can't arbitrate either, since the Z refinement is non-uniform (~4.4 m parents, ~2.8 m children — neither
  ratio 2 nor 4). Without FIPSLOT you must adopt a convention (densest packing), and the L2 LGR becomes 26×44×24 dense instead of 26×44×48 with hole layers. Cell geometry, activity, and results are identical; IJK addressing within the LGR and anything
  CARFIN-shaped differs. The "grid equals the sidecar import" tests would have to be weakened to geometry-equality.

  Bottom line

  ┌──────────────────────────────────────┬─────────────────────────────────────────┬────────────────────────────────────────────────────────────────────┐
  │               Scenario               │               Change size               │                              Fidelity                              │
  ├──────────────────────────────────────┼─────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┤
  │ FIPNEST + FIPSLOT + REFINE (current) │ done                                    │ exact                                                              │
  ├──────────────────────────────────────┼─────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┤
  │ FIPNEST + REFINE, drop FIPSLOT       │ small (~100 lines decoder)              │ exact except hole-layer LGR dims (L2: 24 vs 48 layers)             │
  ├──────────────────────────────────────┼─────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┤
  │ FIPNEST only, strictly nested data   │ moderate (~200 lines)                   │ same as above — REFINE from chain depth is exact for strict chains │
  ├──────────────────────────────────────┼─────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┤
  │ FIPNEST only, DROGON-style data      │ moderate+ (~300 lines incl. clustering) │ LGR decomposition heuristic, hole dims approximate                 │
  └──────────────────────────────────────┴─────────────────────────────────────────┴────────────────────────────────────────────────────────────────────┘

  If the generator will be re-emitting data anyway, my recommendation is the middle row: FIPNEST + REFINE, drop only FIPSLOT. REFINE is a natural simulator region array (it already exists as a sidecar and flows through a deck as easily as FIPNEST does),
  it removes the entire level-inference problem exactly, and the only remaining loss is hole-layer fidelity — which the generator could even avoid by emitting hole-free refinements. FIPNEST-only is viable for your strict-chain data if you accept the
  dense-packing convention for level 2.

  Want me to implement one of these as a fallback mode (proceed with derivation + warnings when the companions are missing, instead of today's warn-and-fall-back-to-sidecars)?

✻ Worked for 2m 17s

❯ What are the size limitation imposed by storing the INIT file? How big can the grids become?                                                                                                                                                                 

● The limits come in tiers — the format itself is generous, and the practical ceiling is ResInsight's memory, not the INIT encoding.

  Hard format limits

  ~2.1 billion flat cells (2³¹−1). This is the real encoding ceiling, hit twice in the same place: FIPNEST values are INTE (32-bit signed) arrays and store the 1-based flat cell index of the parent, so the largest addressable cell is 2,147,483,646; and
  the Eclipse keyword header stores the element count as a 32-bit int, so a single INTE array caps at the same ~2.1 billion entries. The C++ codec uses int for the values consistently with this, and resfo/resdata share the convention.

  Refinement factor ≤ 99 per axis. Not a grid-size limit, but FIPSLOT's decimal packing (1 + offI + 100·offJ + 10000·offK) caps within-parent offsets at 0–99. The codec guards this and drops cells beyond it as unresolved. Real NHG factors are 2–4, so
  there's enormous headroom.

  One caveat at 16.8 million cells (2²⁴) — GRDECL sidecar only. RifEclipseTextFileReader parses GRDECL values as float, so parent indices above 2²⁴ lose exactness if the auto-exported _FIPNEST.grdecl is ever read back through that reader. Today nothing in
  the import does (the import reads the INIT's INTE arrays, which are exact, and the Python script parses the GRDECL with its own integer parser), but it's a trap worth remembering if the .grdecl ever becomes an import source. There's a comment at the
  export site about this.

  File-size overhead — scales with the padded grid, not the active model

  The three full-length arrays cost 12 bytes per flat cell (3 × 4-byte INTE). On DROGON that's +14.5 MB on a 20.5 MB INIT — a 70% growth. The important scaling behavior: nested hybrid grids are massively padded (DROGON is 1.21M flat cells but only 83,755
  active — every other INIT array is active-length), and the flat layout inflates NX with each appended refinement band. So the overhead grows with the bounding box, not the physical model: a 100M-flat-cell NHG with 1M active cells would add ~1.2 GB to
  its INIT. If that regime is ever real, run-length-friendly alternatives (or active-length encoding with a separate channel for the collapsed host slots) would be the fix — but that's a redesign, not a tweak.

  The practical ceiling: ResInsight itself

  Long before 2.1 billion cells, ResInsight's in-memory model is the constraint, and the FIPNEST arrays are a rounding error next to it. During import the codec transiently holds ~40 bytes per flat cell (the three arrays plus the seven synthesized
  OLDIJK/TMP/REFINE vectors), but the grid itself costs far more: full corner geometry (8 nodes per cell), RigCell structures, and every full-length result array at 8 bytes per flat cell per time step. At 100M flat cells, each loaded result time step
  alone is 800 MB. Realistically, flat NHG grids in the tens of millions of cells are workable on a well-equipped workstation; beyond that the whole import path — not the FIPNEST encoding — becomes the problem.

  So: the encoding is safe to ~2.1B cells, comfortable and exact for anything ResInsight can actually load, and the only number to watch operationally is the 12 bytes/flat-cell INIT growth on heavily padded grids.