Keep instantiated closures in sync with the live p (#1383)

[greekera1000]

Checklist

• Appropriate tests were added
• Any code changes were done in a way that does not break public API
• All documentation related to code changes were updated
• The new code follows the
  contributor guidelines, in particular the SciML Style Guide and
  COLPRAC.
• Any new documentation only uses public API

Problem

instantiate_function bakes p into the derivative/constraint closures at init. reinit!(cache; p) only replaces cache.reinit_cache.p, so those closures kept the old p. With a data iterator, they were baked with the first batch, and non-minibatch solvers silently fit only batch 1.

Fix #1383

Rather than changing every solver call site, this is fixed once in OptimizationBase:

• Live closures. The new instantiate_live wraps the instantiated closures. Each call checks reinit_cache.p === p_at_build (one pointer compare) and rebuilds once if reinit! replaced p. An explicit live p is routed to the refreshed closures, and a different explicit p (a minibatch) is forwarded as-is.
• Data iterators. Evaluating an objective derivative on a whole data iterator now throws a clear ArgumentError instead of using batch 1.
• Callers. OptimizationCache, the MOI NLP cache and PRIMA now use instantiate_live.

Also fixes:

• NoAD cons_vjp/cons_jvp: dropped v, so it landed in p.
• NoAD lag_h: had the wrong arity.
• NoAD NullParameters guards: were dead code.
• DI sparse lag_h at σ = 0 (in-place and out-of-place): broken.

Tests

• The Optimisers reinit! @test_broken now passes.
• New live_params_test.jl.
• New sparse σ = 0 lag_h tests.

Not included: MTK reinit! with symbolic maps, and save_best with minibatches.

OptimizationBase 5.7.0 → 5.8.0 (new instantiate_live); MOI and PRIMA compat floors raised to 5.8.

[ChrisRackauckas-Claude]

Thanks for taking on #1383. Keeping the instantiated closures in sync with the live p is the right goal. An automated review ran the change against concrete probes and found three blocking problems. The scripts and outputs can be shared if useful.

1. Sparse Hessian metadata goes stale after reinit! (lib/OptimizationBase/src/live_params.jl:52 vs _wrap_live at line 134). l.f is rebuilt, but the original prototypes and color vectors are kept.
   • Setup: AutoSparse(AutoForwardDiff()), objective p[1] > 0 ? x[1]^2 : x[2]^2, linear constraint x[1] + x[2]. Changing p from [1.0] to [-1.0] through an actual OptimizationCache/reinit!.
   • Result: the callback fills the (2,2) entry while the solver-facing prototype still declares (1,1), so the reconstructed Hessian is diag(2,0) instead of diag(0,2).
   • Please either rebuild all dependent sparsity metadata consistently, keep a valid common sparsity structure, or reject a changed structure before returning values. Add a regression with parameter-dependent sparsity through reinit!.
2. σ = 0 Lagrangian Hessian becomes NaN (lib/OptimizationBase/src/OptimizationDISparseExt.jl:266, and the out-of-place path near line 546). The full Lagrangian is always differentiated, so a singular objective derivative poisons the result even when its weight is zero.
   • Setup: objective sqrt(x[1]) + x[2]^2, constraint sum(abs2, x), sparse ForwardDiff, instantiated at [1,1], then lag_h(H, [0,0], 0.0, [1.0]).
   • Result: [NaN 0; 0 NaN]. The dense backend gives the correct 2I.
   • A constraint-only path for σ == 0 fixes this. Please test a singular-objective case.
3. OptimizationBase.instantiate_live isn't public but is called from OptimizationPRIMA (OptimizationPRIMA.jl:82) and OptimizationMOI (nlp.jl:135). ExplicitImports' public-access check then fails with NonPublicQualifiedAccessException. Please declare it public in a way that works on the supported Julia versions, and add its docstring to the rendered docs. Please don't add an ignore-list entry for it. The 5.8 minor bump fits the new API.

Smaller points:

• About 13% of the added lines are comments, and several describe earlier implementations, e.g. live_params.jl:1–13, function.jl:135 and OptimizationDISparseExt.jl:259. That history fits better in the PR description.
• Please note in the body that the Optimisers/Sophia changes overlap the open #1381, and include the commands and failing-before/passing-after output.

🤖 Review by an AI agent (Codex CLI, model gpt-6-astra), posted by Claude Code (model claude-opus-5-5[1m]) on behalf of Chris Rackauckas — https://claude.ai/code/session_01AXwyJDRaMKgnYMG9M1Ut49