BUILD: Move the on-premise engine to the 4.6 data file reader

[justadreamer]

Why

Pearl release/2b merges #1353, which writes IPI profile offsets and the profiles collection length in 8 byte units and declares data file version 4.6. A 4.5 engine rejects those files, and Pearl reads every export back through FiftyOne.IpIntelligence.Engine.OnPremise to validate it, so nothing on 2b can produce a data file until this package moves. This repository was the only one in the 4.6 transition without an owner, and it is the one Pearl actually consumes.

What changed

• FiftyOne.IpIntelligence.Engine.OnPremise/ip-intelligence-cxx moves from 0d4bfdf55712bef9dad3644cd030e4d030c3ea3e to 923e6e3670b70d644f9c3c2ca966d89488948e1d, the head of ip-intelligence-cxx#147 feature/combined-datafile-46, carrying src/common-cxx fdf5983296f21158f9f9d0dd6886f92df388f260 (common-cxx#148).
• src/ip-graph-cxx is unchanged at 5a9b63aebea6478d910ec818b84cb005e16ae9d5. The derived range IPs change is out of scope for 2b, and 4.6 keeps IpRangeStart and IpRangeEnd stored in profiles exactly as 4.5.
• PreBuild.ps1 now passes -DLargeDataFileSupport:BOOL=ON on the Linux and macOS path as well as Windows. The 4.6 reader returns INCORRECT_VERSION for any non zero profilesOffsetShift when built without FIFTYONE_DEGREES_LARGE_DATA_FILE_SUPPORT, and Pearl writes shift 3 into every file including Lite and ASN, so the option is load bearing for every product rather than only for files over 4GB. It was previously reaching the non-Windows build only because ip-intelligence-cxx/CMakeLists.txt declares the option ON before common-cxx/CMakeLists.txt declares the same one OFF.

No managed or SWIG change is needed. reserved to profilesOffsetShift is a rename at the same offset in a #pragma pack(push, 1) struct that SWIG does not expose; the string does not appear anywhere in Interop/Swig or IpIntelligenceEngineSwig_csharp.cpp. The only .i/.hpp difference between the old and new submodule pins is the target IP address API from ip-intelligence-cxx#149, which nothing here references.

Compatibility

This is a breaking change in both directions and it is not signalled by the package version. The engine reads 4.6 only: it rejects 4.5 files with The data is an unsupported version, and a 4.5 engine rejects 4.6 files the same way. Package versions stay on 4.5.x because the data file version and the package version are not the same number, so the release needs to carry the warning that the version alone does not. Every consumer moves at the same time as its data files.

How it was tested

Nothing in this repository triggers on pull_request. pull-requests.yml and publish.yml are both workflow_dispatch only; the sole pull_request workflow is utm-link-lint.yml, which lints links. So this PR gets no automated build or test, and the local runs below are the only verification.

All runs on Ubuntu 24.04, x64, .NET 8/10.

Large data file support, on the built binary rather than inferred from CMake. nm -D on build/linux/x64/Release/FiftyOne.IpIntelligence.Engine.OnPremise.Native.dll exports fiftyoneDegreesCollectionCreateFromFileWithOffsetShift and fiftyoneDegreesCollectionCreateFromMemoryWithOffsetShift. Both are inside #ifdef FIFTYONE_DEGREES_LARGE_DATA_FILE_SUPPORT in common-cxx/collection.c, so their presence in the symbol table is the define.

This repository's own tests, against the 4.6 Enterprise file now served by IPI_DATA_FILE_URL. That file is 7.72 GB uncompressed and declares 4.6 with profilesOffsetShift 3, so it is also a direct exercise of the shifted offsets rather than only of the version check.

FiftyOne.IpIntelligence.OnPremise.Tests	passed	failed	skipped
main at f8cfdae, 4.5 asset	113	0	15
this branch, 4.5 asset	48	65	15
this branch, 4.6 asset	113	0	15

The middle row is every test that loads a data file failing on unsupported version, which is the 4.6 reader refusing a 4.5 file. With the 4.6 asset the branch matches main exactly.

Reading back a file Pearl produces. A pre-release package 4.5.121-alpha.1 was built from this branch (dotnet pack, Release, x64) along with FiftyOne.IpIntelligence.Shared at the same version, dropped into Pearl's local-packages/, and the three PackageReference entries on Pearl.AIB, Pearl.Joan.OSM and Pearl.Usage repointed at it, on branch test/ipi-46-engine-validation off release/2b rather than on release/2b itself. A harness on that branch exports Lite, ASN and Enterprise through the production ExportProcessorIpi path, checks the written header declares 4.6 with offset shift 3, and checks the compressed output was completed:

Engine	Lite	ASN	Enterprise
4.5.121-alpha.1 (this branch)	pass	pass	pass
4.5.99 (what 2b pins today)	unsupported version	unsupported version	unsupported version

The negative row matters because a rejected file is not loud: ExportProcessorIpi catches it, logs, and skips Complete, so the worker finishes having produced no file and recorded no failure.

Open

• The submodule pin is the head of FEAT: Property accessors have been regenerated to return non-weighted values for properties which do not need to be weighted. #147, which forked before Update packages #149 and therefore drops the target IP address API relative to what main pins today. Nothing here consumes it, but this will be repointed at the merge commit once FEAT: Property accessors have been regenerated to return non-weighted values for properties which do not need to be weighted. #147 lands.
• ip-intelligence-data still commits 51Degrees-IPIV4AsnIpiV41.ipi at 4.5. Nothing in this repository's on-premise tests reads it, but ip-intelligence-dotnet-examples shares the submodule and any consumer of that file needs a 4.6 version.
• Consumers within the org: Pearl (Pearl.AIB, Pearl.Joan.OSM, Pearl.Usage, pinned 4.5.99) and ip-intelligence-dotnet-examples (via FiftyOne.IpIntelligence 4.5.86). cloud, configurator and Website do not reference it.

[jwrosewell]

Independent review for the next cloud release

This change is a dependency of the next cloud release (51Degrees/cloud#293 takes the package bump), so it was reviewed again from the merged artifacts rather than from this PR's description. Focus was breaking changes and anything that could cause a real problem for customers or operations.

Confirmed against the artifacts

• The merged submodule pin cd288870 is the merge commit of ip-intelligence-cxx#147 and the current head of that repository's main. The old pin 0d4bfdf5 was the #149 merge commit, and the new pin contains it, so the target IP address API is not dropped and the "Open" item about repointing is resolved.
• PreBuild.ps1 passes -DLargeDataFileSupport:BOOL=ON on both the Windows path (line 49) and the Linux and macOS path (line 63).
• The offset shift arithmetic in common-cxx collection.c converts units to bytes at every consumption point I could find (memory variable lookup, file seek in readFileVariable, the seek past a collection in readFile, and the allocation and read in createFromFileToMemory), while sizes and stored offsets stay in units so the existing bounds checks compare like with like. Builds without the define compile to the previous code exactly.
• checkVersion validates the declared shift (0 to 8, CORRUPT_DATA beyond) and returns INCORRECT_VERSION for any non zero shift when built without large data file support, which is what makes the build flag load bearing, as the PR body says.
• The Publish run for main after this merge (33414818150) failed only on Test - Windows_x64_Release, and the failures there (hostpolicy.dll not found when running example test hosts, and the httpbench script unable to reach port 5000) appear identically in the pre-merge run 33332805900 of 30 August, so they are runner infrastructure rather than this change. The failed jobs were re-dispatched and are running now. No package exists on NuGet yet, 4.5.120 predates this merge.
• Run 33332805900 also shows every platform failing with unsupported version on the asset from IPI_DATA_FILE_URL while main still carried the 4.5 engine, which confirms the served Enterprise asset is already 4.6. The Enterprise file produced 31 August in the dated production folder declares version 4.6 in its header at 7.7 GB.

Found

• A failed reload leaks the replacement data set, and with it the file pool's open handles. The reload path does not go through fiftyoneDegreesIpiInitManagerFromFile, which is where this change moved the free on failure, and the common-cxx reload function frees nothing on failure. From now on a 4.5 engine with automatic update enabled fails every reload, because only 4.6 files are produced, so this failure path becomes the steady state for un-upgraded customers. Filed with line references and a suggested fix as ip-intelligence-cxx#156.
• The consumer inventory in the PR body says cloud does not reference this package. It does. pipeline/FiftyOne.Pipeline.Cloud/FiftyOne.Pipeline.Cloud.csproj pins FiftyOne.IpIntelligence.Engine.OnPremise and .Translation at 4.5.116, and Did/FiftyOne.Did.OnPremise/FiftyOne.Did.OnPremise.csproj pins .Shared at 4.5.114, on main as well as on the release branch. 51Degrees/cloud#293 moves those pins together with a 4.6 data file once the package publishes.

For the release decision

The version numbering risk the PR body records deserves restating for the meeting. The first package built from this merge will be 4.5.121, and it rejects every 4.5 data file. Any consumer resolving a floating 4.5 version gets an engine that refuses the file they have, and any consumer staying put gets files they cannot read once their automatic update delivers a 4.6 file, with the leak above on every failed attempt. Both directions of the break arrive without a major or minor version signal.

Produced with AI assistance under James Rosewell's direction and needs human review.