Skip to content

Test worker isolation after missing-payload downloads - #329

Merged
Bernd Verst (berndverst) merged 3 commits into
mainfrom
andystaples-missing-payload-isolation
Nov 10, 2026
Merged

Bernd Verst (berndverst) merged 3 commits into
mainfrom
andystaples-missing-payload-isolation

Conversation

@andystaples

Copy link
Copy Markdown
Contributor

Summary

Add two large-payload regression cases using the existing in-memory backend, fake payload store, and real Python gRPC worker:

  • Remove a payload referenced by an orchestration or activity delivery and observe the affected work item's abandonment.
  • Keep two unrelated large-payload activities in progress during the failure.
  • Verify fresh work completes before releasing the held activities, then verify both original orchestrations complete with their expected outputs.

The tests use synchronization events rather than timing-dependent sleeps. They run in the existing core test suite without Docker, Azure Storage, a Functions host, or additional CI jobs.

Scope

Existing abandonment behavior is unchanged. The missing payload is restored after abandonment to bound immediate orchestration redelivery; these tests do not assert a new terminal-failure or retry policy.

This is native Python SDK worker isolation coverage, not Functions-host or DTS-provider isolation coverage. It does not validate host-managed payload externalization or the upstream fix for that path.

Closes #312 as superseded by this narrower SDK regression coverage; the proposed DTS-host test matrix is intentionally not implemented.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 2, 2026 19:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The regression test is timing-dependent because the healthy activity’s 10-second timeout spans two sequential waits that each independently allow 10 seconds, so valid execution can still fail.

1 open issue
What changed in this PR

Adds native Python SDK worker regression coverage for missing-payload download isolation, without changing abandonment behavior. This is narrower than the Functions-host/DTS coverage proposed in #312.

Changes:

  • Tests missing payloads in both orchestration and activity deliveries.
  • Verifies fresh work completes while two unrelated activities remain blocked, then checks their expected outputs.
  • Restores missing payloads after abandonment to bound immediate redelivery.
File Description
tests/​durabletask/​test_large_payload.py Adds parameterized isolation tests using the in-memory backend, fake payload store, and real gRPC worker.

Review effort: Balanced

Comment thread tests/durabletask/test_large_payload.py Resolved

@berndverst Bernd Verst (berndverst) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The added cases exercise the intended native-worker missing-download paths and provide useful isolation coverage without changing runtime behavior, performance, or public compatibility. I independently confirmed the existing timeout concern: a six-second missing-download/abandonment stage followed by a six-second fresh activity causes both parameterized cases to fail their original healthy orchestrations even though each controller stage is within its ten-second timeout. Please remove the healthy activity's independent ten-second release deadline and rely on the bounded controller waits plus finally-based release before merging. The broader Functions/DTS acceptance criteria from #312 remain outside this coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

@berndverst Bernd Verst (berndverst) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The held-activity timeout race is fixed. I reran the original reproduction with a six-second missing-download delay followed by a six-second fresh-activity delay; both parameterized cases now pass while preserving the isolation and output assertions. Controller-failure cleanup also releases the held activities correctly. No remaining actionable findings in the fix; the coverage remains scoped to native Python SDK worker isolation.

@berndverst
Bernd Verst (berndverst) merged commit 9d5974a into main Nov 10, 2026
25 checks passed
@berndverst
Bernd Verst (berndverst) deleted the andystaples-missing-payload-isolation branch November 10, 2026 01:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Azure Functions] Add DTS coverage for isolation of SDK-managed missing-payload failures

3 participants