Repository navigation
Fix replay determinism for retry timeouts - #326
Conversation
Use orchestration time for activity and sub-orchestration retry expiration, normalize UTC timestamps, and cover replay and deadline boundaries. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
The datetime checks mishandle timestamps whose tzinfo is non-None but utcoffset() returns None, causing host-dependent conversions that can prematurely exhaust retry timeouts.
1 open issue
What changed in this PR
This PR addresses #316 by using replay-safe orchestration time for activity and sub-orchestration retry timeouts, preventing delayed history replay from incorrectly suppressing retries.
Changes:
- Passes logical time into retry calculations and normalizes timezone-aware timestamps while retaining the direct-call wall-clock fallback.
- Adds regression coverage for replay, timeout boundaries, timezone handling, and successful completion after retries.
- Documents the fix and upgrade precautions across all three packages without changing provider versions or dependency minimums.
| File | Description |
|---|---|
| tests/durabletask/test_retry_timeout.py | Adds timeout and replay regression tests. |
| durabletask/worker.py | Supplies orchestration time to both retry paths. |
| durabletask/task.py | Adds explicit retry time and timestamp normalization. |
| durabletask-azuremanaged/CHANGELOG.md | Documents the provider impact and core upgrade guidance. |
| CHANGELOG.md | Records the fix and warns about existing history compatibility. |
| azure-functions-durable/CHANGELOG.md | Documents the provider impact and core upgrade guidance. |
Review effort: Balanced
Verify expiration on the fifth failure after the 14-second deadline, with a higher attempt limit so timeout exhaustion is exercised independently. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve released changelog entries and keep retry timeout corrections under Unreleased in all affected packages. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Guard UTC conversion with utcoffset and cover undefined offsets on both retry timestamps without relying on the host timezone. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Bernd Verst (berndverst)
left a comment
There was a problem hiding this comment.
No blocking findings. The activity and sub-orchestration retry paths now use the replay-safe orchestration clock, preserving timeout-boundary, attempt-limit, backoff and retry-timer ordering behavior. I also checked the inherited Azure Managed and Functions worker paths. The documented upgrade warning is important: histories that previously caught a wall-clock retry timeout can diverge under the corrected behavior, so affected in-flight instances must be drained on the old SDK before rollout. Subject to that rollout requirement, this is a net-positive fix.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Bernd Verst (berndverst)
left a comment
There was a problem hiding this comment.
Re-reviewed d44f7f7 after merging current main. No blocking findings. The conflict resolution retains both sets of changelog entries and replay-upgrade warnings; the replay-safe retry implementation and undefined-offset handling are unchanged. Retry and serialization replay regressions pass together, and Azure Managed/Functions worker checks preserve timer ordering and timeout-boundary behavior. The documented drain-before-upgrade requirement still applies to affected histories.

Summary
Activity and sub-orchestration retry timeout decisions now use the orchestration's replay-safe logical clock instead of wall-clock time. Replaying a historical failure after the real-time deadline no longer incorrectly suppresses its retry.
RetryableTask.compute_next_delay()accepts an optional current timestamp, normalizes timezone-aware inputs to UTC, and treats naive inputs as UTC. Direct calls without a timestamp retain their wall-clock fallback. Retry attempt limits, backoff, interval caps, and the existing expiration boundary are preserved; an absent timeout remains unbounded.Regression coverage exercises activity and sub-orchestration history replay, recorded retry timers through successful completion, deadline boundaries, timezone offsets, and calls without an explicit timestamp.
Upgrade and release coordination
Histories that recorded timeout failures based on wall-clock time can replay differently under the corrected behavior, including orchestrations that caught those failures. Drain affected instances on the previous SDK before upgrading. This change does not recover already-failed instances.
The core, Azure Managed, and Azure Functions changelogs document the fix. Provider versions and dependency minimums remain unchanged; coordinate provider releases after the corrected core SDK is released.
Fixes #316