[Azure Functions] Rewind E2E intermittently remains Failed after accepted rewind #331

Failure

tests/azure-functions-durable/e2e/test_v1_client_e2e.py::test_rewind intermittently times out waiting for a rewound orchestration to complete.

Observed on #326, commit 4b246630448f7b6ee3240a5935ca0d50ca4ad541:

• Failed strict-typing=1 job
• Ubuntu runner, Python 3.13.17; nox -s functions_e2e
• Result: 1 failed, 83 passed, 6 skipped. The strict-typing=0 E2E job passed, including this test.
• Host logs are attached to the run as func-host-logs-strict-1, particularly v1_style/_func_host.log.

The test starts rewind_target, observes Failed, posts to /api/rewind/{instance_id}, and receives HTTP 202. It then waits 60 seconds for Completed, but the instance remains Failed with the activity's deliberate first-attempt failure:

TimeoutError: Orchestration 14d366939d544d27988e1e0a0197d700 did not reach 'Completed' within 60s
runtimeStatus: Failed
output: durabletask.task.TaskFailedError: ... Activity task #1 failed: failing on first attempt (rewind to retry)

Host evidence

For instance 14d366939d544d27988e1e0a0197d700, the captured host log shows:

• 19:43:23.936Z: client requests rewind.
• 19:43:23.945Z and .946Z: rewind reads report one history event.
• 19:43:23.950Z: instance status is updated to Pending.
• 19:43:23.955Z: host reports the orchestration was rewound.
• 19:43:23.966Z: orchestration schedules fail_once#1.
• 19:43:23.984Z: fail_once raises its deliberate first-attempt error.
• 19:43:24.014Z: instance status is updated to Failed and stays there through polling.

The sample activity uses an in-process per-token attempt counter. These observations warrant investigating history/status visibility around the initial failure and rewind, together with activity counter lifetime. The exact race or underlying defect is not yet established; the log excerpt alone does not prove which component is responsible.

Scope

This differs from #330, which is an assertion failure caused by adjacent host log records in the logging E2E test. Here the assertion is an orchestration status timeout, not log formatting.

rewind_target calls fail_once without a retry policy, so it does not exercise the RetryableTask.compute_next_delay() changes in #326. Core and Azure Managed CI passed across all Python versions on this commit.

Expected behavior

Once the deliberately failed activity is rewound, the next invocation should succeed and the orchestration should reach Completed. Investigate and make the test/runtime interaction reliable without concealing persistent failures with unconditional retries or longer arbitrary sleeps.