Skip to content

Allow LAs to wake up workflow while another is still executing - #1442

Merged
Sushisource merged 6 commits into
mainfrom
sj/fix-la-concurrent-resolution
Aug 4, 2026
Merged

Sushisource merged 6 commits into
mainfrom
sj/fix-la-concurrent-resolution

Conversation

@Sushisource

@Sushisource Sushisource commented Aug 3, 2026

Copy link
Copy Markdown
Member

What was changed

Live execution

Previously, when an activation scheduled local activities, ManagedRun held lang's activation
completion response until every outstanding LA finished or the workflow task heartbeat fired. LA
results were applied internally, but lang could not poll another activation, so one long-running LA
could block a concurrent sequence of short LAs.

Core now acknowledges that activation completion with DoNothing while keeping the WFT and its
heartbeat open. Each LA result is applied and immediately checked for a new activation, even while
other LAs remain outstanding:

LA result arrives -> apply result -> emit activation immediately -> lang schedules more work

When the final LA finishes, Core completes the WFT normally with the accumulated marker commands.
Heartbeat completions request another WFT when LAs remain, and zero-sized-cache workers retain the
run until this process finishes.

The live LA state-machine path is unchanged:

Executing --Schedule--> RequestSent --HandleResult--> MarkerCommandCreated --CommandRecordMarker--> ResultNotified --MarkerRecorded--> MarkerCommandRecorded

The state machine already produced a resolution from HandleResult; the live-path bug was that
ManagedRun prevented that resolution activation from reaching lang while another LA was running.

Replay compatibility

Incremental live delivery can create different activation groupings, so replay must reproduce the
order recorded by LA markers. Previously, new_local_activity could construct an LA directly in
ReplayingPreResolved by looking up its sequence number, causing its Schedule transition to emit
the resolution immediately. Results were therefore applied as activities were scheduled rather than
strictly in marker order.

Now every replaying LA starts unresolved and follows:

Replaying --Schedule--> WaitingResolveFromMarkerLookAhead --HandleKnownResult--> ResolvedFromMarkerLookAheadWaitingMarkerEvent --MarkerRecorded--> MarkerCommandRecorded

workflow_machines.rs retains peeked results in marker order in a VecDeque. After each workflow
iteration, it supplies HandleKnownResult from the front of that queue while the corresponding LA
machine exists, stopping at the first marker for an LA the workflow has not scheduled yet. The LA
state machine still owns one activity's lifecycle and validation; workflow_machines.rs owns the
cross-activity ordering needed to preserve activation boundaries.

This reproduces both old batched histories and new incrementally delivered histories without an SDK
compatibility flag. The old pre-resolved states and fake-marker plumbing are no longer needed.

Why?

This allows a sequence of short local activities to continue making progress while a long-running
local activity executes in parallel, without introducing nondeterminism when existing histories are
replayed.

Checklist

  1. Closes: [Feature Request] Enable notifying lang in realtime of local activity resolutions #856 & [Bug] NDE replaying nested promises sdk-typescript#1744

  2. How was this tested:

    • Added a live integration test proving the short LA sequence completes before the gated long LA,
      then fetched and replayed the generated history.
    • Added a canned-history replay test for the batched marker ordering produced by the old behavior.
    • Ran the full local activity integration module: 84 passed.
    • Ran cargo test -p temporalio-sdk-core, cargo lint, and cargo test-lint.
    • Regenerated LocalActivityMachine_Coverage.puml.
  3. Any docs updates needed?

    No. An Unreleased changelog entry is included.

@Sushisource
Sushisource force-pushed the sj/fix-la-concurrent-resolution branch from 1bd838e to 478f379 Compare August 4, 2026 00:06
@Sushisource
Sushisource marked this pull request as ready for review August 4, 2026 00:06
@Sushisource
Sushisource requested a review from a team as a code owner August 4, 2026 00:06
@Sushisource
Sushisource force-pushed the sj/fix-la-concurrent-resolution branch from a63386c to d2a8ee4 Compare August 4, 2026 17:29
@mjameswh

mjameswh commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

It looks like this PR also covers the bug that was addressed by #1075. Do you confirm?

Comment thread crates/sdk-core/src/worker/workflow/managed_run.rs Outdated
Comment thread crates/sdk-core/src/worker/workflow/machines/workflow_machines.rs
@Sushisource

Copy link
Copy Markdown
Member Author

It looks like this PR also covers the bug that was addressed by #1075. Do you confirm?

Yep & added tests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request] Enable notifying lang in realtime of local activity resolutions

3 participants