Skip to content

Only the end-of-stream sentinel may stop the convergence output thread - #7267

Merged
bska merged 2 commits into
OPM:masterfrom
hnil:fix-convergence-output-crash
Sep 17, 2026
Merged

bska merged 2 commits into
OPM:masterfrom
hnil:fix-convergence-output-crash

Conversation

@hnil

@hnil hnil commented Aug 2, 2026 •

Copy link
Copy Markdown
Member

#7257 has landed since this was opened and fixes the segfault it originally targeted. What is left is the semantics underneath that crash.

OutputRequest used "reports is empty" as the shutdown sentinel, but SimulatorConvergenceOutput::write() builds one request per StepReport — so a step that aborted before its first convergence check produces an ordinary request with no reports, indistinguishable from the sentinel. The writer thread then stops for the rest of the run. This makes the sentinel an explicit isFinal flag, lays the header out from the first request that actually carries a report, and skips report-less requests instead of dereferencing them. endThread() is idempotent and a destructor backstops it.

Checked on current master: wconprod/WCONPROD-01.DATA with --output-extra-convergence-info=steps,iterations exits cleanly, and its INFOITER is empty both with and without this PR — that run never completes a step, so there is nothing to write. The "zero-byte INFOITER" claim in the original description does not hold; INFOITER is flushed incrementally as steps complete. I have left that out.

@hnil
hnil requested a review from bska August 4, 2026 12:46
@hnil hnil added the manual:irrelevant This PR is a minor fix and should not appear in the manual label Aug 7, 2026
@hnil
hnil force-pushed the fix-convergence-output-crash branch from 9093b40 to 9ebcef1 Compare August 11, 2026 08:28
@hnil
hnil marked this pull request as ready for review August 11, 2026 08:49
@hnil hnil changed the title Fix crash and data loss in convergence output on failed runs Only the end-of-stream sentinel may stop the convergence output thread Aug 11, 2026
@hnil
hnil force-pushed the fix-convergence-output-crash branch from 9ebcef1 to 91524cb Compare August 11, 2026 12:06
@bska
bska requested a lite review from Copilot August 18, 2026 07:57
@bska

bska commented Aug 18, 2026

Copy link
Copy Markdown
Member

jenkins build this please

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refines the lifecycle and shutdown semantics of the asynchronous convergence output writer to ensure only an explicit end-of-stream sentinel can terminate the writer thread, avoiding premature shutdown when a valid step produces an empty convergence report sequence.

Changes:

  • Introduces an explicit isFinal flag on OutputRequest and updates the writer loop to terminate only on that sentinel (not on “empty reports”).
  • Makes INFOITER header generation resilient by deriving columns from the first request that actually contains a convergence report, and skipping report-less requests.
  • Adds shutdown backstops by making endThread() idempotent and ensuring the convergence output thread is joined during destruction paths.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
opm/simulators/flow/SimulatorFullyImplicit_impl.hpp Flushes any remaining step reports on simulator destruction and ends the convergence output thread.
opm/simulators/flow/SimulatorConvergenceOutput.hpp Adds a destructor that calls endThread() as a safety backstop.
opm/simulators/flow/SimulatorConvergenceOutput.cpp Makes endThread() idempotent by resetting the stored thread after joining.
opm/simulators/flow/ExtraConvergenceOutputThread.hpp Extends OutputRequest with an explicit isFinal end-of-stream marker.
opm/simulators/flow/ExtraConvergenceOutputThread.cpp Updates writer logic to use isFinal sentinel, defers header creation until a real report exists, and skips empty-report requests.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread opm/simulators/flow/ExtraConvergenceOutputThread.cpp
Comment thread opm/simulators/flow/SimulatorConvergenceOutput.cpp
@bska

bska commented Aug 27, 2026

Copy link
Copy Markdown
Member

Would you mind responding to Copilot's review comments, please?

hnil and others added 2 commits September 17, 2026 15:47
When a simulation aborted before completing any report step, the
convergence output thread crashed with SIGSEGV: the end-of-stream marker
enqueued by signalLastOutputRequest() is a request with an empty reports
vector, and writeConvergenceHeader() unconditionally dereferenced
reports.front() of the first request to lay out the metric columns.

Three related problems fixed:

- The shutdown sentinel is now an explicit isFinal flag on OutputRequest
  instead of "reports is empty". An ordinary request with no convergence
  reports (a step that aborted before its first convergence check) was
  previously indistinguishable from the sentinel and silently shut the
  writer thread down mid-run.

- The header is laid out from the first request that actually carries a
  convergence report, and requests without reports are skipped instead
  of dereferenced.

- The step reports accumulated up to the point of failure are now
  flushed to the INFOITER file before the writer thread is joined, and
  endThread() is idempotent (also called from a new
  SimulatorConvergenceOutput destructor as a backstop). Previously a
  failed run left a zero-byte INFOITER -- discarding the convergence
  history of exactly the runs whose convergence history is most needed.

Reproducer: any deck that fails in its first report step with
--output-extra-convergence-info=steps,iterations, e.g. opm-tests
wconprod/WCONPROD-01.DATA (crash) or spe02/SPE02-SPIDER-01.DATA
(zero-byte INFOITER); both now exit cleanly with diagnostics on disk.
Healthy runs produce byte-identical INFOITER content.
- endThread() joins only a joinable thread.
- Regression test: a request with no reports that is not the sentinel
  must not stop the output thread.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@hnil
hnil force-pushed the fix-convergence-output-crash branch from 91524cb to b4bcbda Compare September 17, 2026 14:21
@hnil

hnil commented Sep 17, 2026

Copy link
Copy Markdown
Member Author

jenkins build this please

@bska bska left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for the updates. This looks good to me now and as the build check is green, I'll merge this into the master branch to enable the new behaviour.

@bska
bska merged commit 5a53d32 into OPM:master Sep 17, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

manual:irrelevant This PR is a minor fix and should not appear in the manual

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants