[ChrisWaschl]

Summary

fmp_read_values silently drops values in fp3/fp5 files when their field reference is two bytes long. handle_chunk_read_values_v3 only accepts FMP_CHUNK_FIELD_REF_SIMPLE, and process_block_v3 turns only one-byte references into that type. Two kinds of values are affected:

• Columns with an index of 128 and above. Their values sit at [5].[record] with a two-byte reference (chunk code 0x02) encoded like a path component, e.g. 0x80 0x8D for column 269. path_value decodes it to the same index fmp_list_columns reads from [3].[5].[269]. Records whose values all live in such columns are missing from the output entirely.
• Segments of long values with a two-byte segment number, e.g. 0x00 0x01 at [5].[record].[column]. The column comes from the path here, and process_value already handles these chunks; they just never reached it.

Column names and column count are correct and nearly all records appear, so the loss only shows up when checking the content.

Changes

• handle_chunk_read_values_v3 also accepts FMP_CHUNK_FIELD_REF_LONG with a two-byte reference. Longer references are still skipped, since nothing here shows what they mean.
• process_value decodes the column index from a two-byte reference at the record level when its high bit is set, using path_value.
• HACKING documents both encodings in the fp5 section.
• CI exports CAKUNITS.FP5 and checks for a description that was missing before.

Testing

• Test files: JSON output is byte-identical for all files in test/data except CAKUNITS.FP3 and CAKUNITS.FP5. These keep their 1,351 records and get 84 more values in 57 records (e.g. Description 1,231 → 1,277); no existing value changes. Files that failed before still fail the same way.
• Columns ≥ 128: The test files have no table with more than 127 columns, so this case was checked with 104 private fp5 files (13 GB). Each was exported with fmp2sqlite with and without the patch:
  • 24 files get more values, the other 80 are unchanged. Across all files, the number of values grows from 177.8 million to 424.8 million. In a 780 MB file with 191 columns, 46 columns had values before and 138 after.
  • 4 files get 173 records that were missing before. Every previous row is still there, in the same order and with its values unchanged.
  • All exports exit with 0, and no value contains control characters other than tab, CR, LF and the repetition separator.
• Column assignment: For a 251 KB file and the 780 MB file, the new values were compared with the raw fmpdump output: every value that fmpdump prints as text ends up in the column defined at [3].[5].[index]. The 251 KB file shows the boundary: column 127 was read before, column 128 was not.
• The library builds with -Wall -Werror -pedantic-errors.

Open question

The rich-text filter in process_value skips segments with key 0 only for one-byte references. A two-byte key 0x00 0x00 would therefore be read as text. It does not occur in the test files or in the two private files checked with fmpdump, so I did not add a filter for it rather than guess at its meaning.

[Copilot]

🟢 Approval recommended

The narrowly scoped changes reuse existing parser logic, include a regression check, and have no identified blocking issues.

No open issues 🎉

What changed in this PR

Fixes silently omitted fp3/fp5 values by allowing the value reader to handle two-byte field references through existing decoding and assembly logic.

Changes:

• Reads values for column indexes ≥128 and long-value segments with two-byte references.
• Documents both reference encodings in the fp5 format notes.
• Adds a CAKUNITS.FP5 regression check to Linux and macOS CI.

File	Description
src/​read_values.c	Accepts two-byte references and decodes record-level column indexes.
HACKING	Documents column-index and long-value segment encodings.
.github/​workflows/​build.yml	Checks that an FP5 export includes previously missing text.

_{Review effort: Balanced}