Skip to content

fix(core): delay zero-cache eviction until LA resolutions drain - #1482

Merged
jmaeagle99 merged 4 commits into
mainfrom
fix/la-resolution-lost-on-evict
Aug 11, 2026
Merged

jmaeagle99 merged 4 commits into
mainfrom
fix/la-resolution-lost-on-evict

Conversation

@jmaeagle99

@jmaeagle99 jmaeagle99 commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

What was changed

The zero-sized-cache eviction in WFStream::process_completion now also skips runs that still have jobs queued for lang, not just runs still waiting on local activities. This defers the eviction rather than skipping it since process_completion is called for every completion; the last completion will not have pending jobs nor outstanding LAs, thus the eviction is queued.

Why?

With max_cached_workflows = 0 and local activities, a run could be evicted mid-workflow-task, stranding its LA markers and livelocking the workflow.

#1442 delivers LA resolutions incrementally, so a resolution arriving while an earlier one is still outstanding with lang is parked as a pending job. That creates a state which didn't exist before that change: no outstanding LAs, but resolutions still undelivered. So the eviction landed mid-task: the accumulated RecordMarker commands were
discarded with the run and the WFT was never reported. Task timeout occurs and server redelivers the WFT; the same run is replayed, same local activities, same result, repeated indefinitely.

On debug builds, like in CI, a panic is intentionally raised with the message There should not be any outgoing commands or messages when preparing a completion response if the activation was only an eviction. This is an SDK bug..

Checklist

  1. How was this tested: New explicitly-driven core test and locally with sdk-python submodule update

  2. Any docs updates needed? No

@jmaeagle99
jmaeagle99 marked this pull request as ready for review August 11, 2026 18:14
@jmaeagle99
jmaeagle99 requested a review from a team as a code owner August 11, 2026 18:14
@jmaeagle99
jmaeagle99 enabled auto-merge (squash) August 11, 2026 20:54
@jmaeagle99
jmaeagle99 merged commit 5db6293 into main Aug 11, 2026
21 checks passed
@jmaeagle99
jmaeagle99 deleted the fix/la-resolution-lost-on-evict branch August 11, 2026 20:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants