Skip to content

Fix replay determinism for retry timeouts - #326

Merged
Bernd Verst (berndverst) merged 6 commits into
mainfrom
andystaples-retry-timeout-determinism
Nov 10, 2026
Merged

Bernd Verst (berndverst) merged 6 commits into
mainfrom
andystaples-retry-timeout-determinism

Conversation

@andystaples

Copy link
Copy Markdown
Contributor

Summary

Activity and sub-orchestration retry timeout decisions now use the orchestration's replay-safe logical clock instead of wall-clock time. Replaying a historical failure after the real-time deadline no longer incorrectly suppresses its retry.

RetryableTask.compute_next_delay() accepts an optional current timestamp, normalizes timezone-aware inputs to UTC, and treats naive inputs as UTC. Direct calls without a timestamp retain their wall-clock fallback. Retry attempt limits, backoff, interval caps, and the existing expiration boundary are preserved; an absent timeout remains unbounded.

Regression coverage exercises activity and sub-orchestration history replay, recorded retry timers through successful completion, deadline boundaries, timezone offsets, and calls without an explicit timestamp.

Upgrade and release coordination

Histories that recorded timeout failures based on wall-clock time can replay differently under the corrected behavior, including orchestrations that caught those failures. Drain affected instances on the previous SDK before upgrading. This change does not recover already-failed instances.

The core, Azure Managed, and Azure Functions changelogs document the fix. Provider versions and dependency minimums remain unchanged; coordinate provider releases after the corrected core SDK is released.

Fixes #316

Use orchestration time for activity and sub-orchestration retry expiration, normalize UTC timestamps, and cover replay and deadline boundaries.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 2, 2026 19:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The datetime checks mishandle timestamps whose tzinfo is non-None but utcoffset() returns None, causing host-dependent conversions that can prematurely exhaust retry timeouts.

1 open issue
What changed in this PR

This PR addresses #316 by using replay-safe orchestration time for activity and sub-orchestration retry timeouts, preventing delayed history replay from incorrectly suppressing retries.

Changes:

  • Passes logical time into retry calculations and normalizes timezone-aware timestamps while retaining the direct-call wall-clock fallback.
  • Adds regression coverage for replay, timeout boundaries, timezone handling, and successful completion after retries.
  • Documents the fix and upgrade precautions across all three packages without changing provider versions or dependency minimums.
File Description
tests/​durabletask/​test_retry_timeout.py Adds timeout and replay regression tests.
durabletask/​worker.py Supplies orchestration time to both retry paths.
durabletask/​task.py Adds explicit retry time and timestamp normalization.
durabletask-azuremanaged/​CHANGELOG.md Documents the provider impact and core upgrade guidance.
CHANGELOG.md Records the fix and warns about existing history compatibility.
azure-functions-durable/​CHANGELOG.md Documents the provider impact and core upgrade guidance.

Review effort: Balanced

Comment thread durabletask/task.py Resolved
Verify expiration on the fifth failure after the 14-second deadline, with a higher attempt limit so timeout exhaustion is exercised independently.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
andystaples and others added 2 commits November 3, 2026 10:17
Preserve released changelog entries and keep retry timeout corrections under Unreleased in all affected packages.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Guard UTC conversion with utcoffset and cover undefined offsets on both retry timestamps without relying on the host timezone.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

@berndverst Bernd Verst (berndverst) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings. The activity and sub-orchestration retry paths now use the replay-safe orchestration clock, preserving timeout-boundary, attempt-limit, backoff and retry-timer ordering behavior. I also checked the inherited Azure Managed and Functions worker paths. The documented upgrade warning is important: histories that previously caught a wall-clock retry timeout can diverge under the corrected behavior, so affected in-flight instances must be drained on the old SDK before rollout. Subject to that rollout requirement, this is a net-positive fix.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

@berndverst Bernd Verst (berndverst) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed d44f7f7 after merging current main. No blocking findings. The conflict resolution retains both sets of changelog entries and replay-upgrade warnings; the replay-safe retry implementation and undefined-offset handling are unchanged. Retry and serialization replay regressions pass together, and Azure Managed/Functions worker checks preserve timer ordering and timeout-boundary behavior. The documented drain-before-upgrade requirement still applies to affected histories.

@berndverst
Bernd Verst (berndverst) merged commit b27a325 into main Nov 10, 2026
25 checks passed
@berndverst
Bernd Verst (berndverst) deleted the andystaples-retry-timeout-determinism branch November 10, 2026 17:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RetryableTask.compute_next_delay uses wall-clock time causing non-deterministic failures on replay with retry_timeout

3 participants