Skip to content

ci: add nightly contract tests with long-running tests enabled - #380

Open
tanderson-ld wants to merge 1 commit into
mainfrom
ta/SDK-3318/dotnet-nightly-long-running
Open

tanderson-ld wants to merge 1 commit into
mainfrom
ta/SDK-3318/dotnet-nightly-long-running

Conversation

@tanderson-ld

@tanderson-ld tanderson-ld commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Requirements

  • I have followed the repository's pull request submission guidelines
  • I have added test coverage for new or changed functionality — n/a, CI configuration only
  • I have validated my changes against all supported platform versions — n/a, the nightly runs a single target by design

Related issues

Tracked in Jira under the RETRY-conformance epic. The issue key is omitted from this body and commit deliberately: the ticket also requires the FDv2 failures below to be tracked, and that follow-up ticket does not exist yet, so an auto-close on merge would be premature.

Counterparts already merged: java-core PR 212 and go-server-sdk PR 454.

Describe the solution you've provided

The harness gates its RETRY-conformance and FDv2 long-running tests behind -enable-long-running-tests, because they wait on real backoff timing and take about an hour. Per-PR CI does not pass it, so none of them have ever run for this SDK — even though the contract test service has declared retry-conformance-fdv1-streaming and retry-conformance-fdv1-polling since the RETRY work landed.

Two changes:

  • .github/actions/contract-tests/action.yml — new enable_long_running_tests input, appended to the params of both harness legs. It defaults to 'false', so sdk-server-ci.yml and sdk-client-ci.yml render byte-identical params to today.
  • .github/workflows/sdk-server-nightly.yml — new. schedule at 04:00 UTC plus workflow_dispatch, single runner, timeout-minutes: 120, test service log uploaded on always().

Warning

This job is expected to fail on three tests, all on the FDv2 leg:

streaming/fdv2/recoverable fallback to secondary synchronizer
streaming/fdv2/recoverable fallback with recovery
streaming/fdv2/permanent fallback with recovery

These are deliberately not suppressed. A nightly that goes red on a real defect is the point — that is how the equivalent Go defect was found and fixed. More than three failures means something else broke.

Worth knowing: these are the same three tests java-core suppresses in test-supressions-fdv2.txt, labelled there as "pending spec alignment between Go and Java." That label looks wrong — go-server-sdk passes all three, so this is a shared Java/.NET implementation gap rather than a spec disagreement. A follow-up ticket covering both SDKs is pending.

Describe alternatives you've considered

  • Suppressing the three FDv2 failures so the job is green from day one. Rejected: it defeats the purpose, and java-core is the cautionary case — the same three have been suppressed there since February and never revisited.
  • A standalone nightly workflow duplicating the build and launch steps instead of extending the shared action. Rejected: the dotnet build/launch sequence is non-trivial and already lives in the composite; an additive input with a safe default keeps it in one place.
  • JUnit XML artifacts, as the Java and Go nightlies emit. Dropped: -junit is a harness flag unrelated to the .NET toolchain, GitHub Actions does not parse an uploaded JUnit file without a separate reporter action, and the harness already prints failing test names to the job log.

Additional context

Verified locally against the real service (dotnet-server-sdk 8.17.2.0), replaying the exact params the composite renders:

  • v2 leg (harness v2.41.0): all tests passed, 4912 total, 17 skipped, 4895 ran.
  • v3 leg (harness v3.2.0-alpha.13): 4930 total, 12 skipped, 4918 ran, 3 failures — the three above.
  • Zero use -enable-long-running-tests to run skip lines on either leg, confirming the flag reaches both.

So the RETRY-conformance work in this SDK passes in full; the failures are confined to FDv2 synchronizer fallback and recovery.


Note

Overview
Adds a nightly Server SDK contract-test workflow and a way to turn on harness tests that wait on real retry backoff (~1 hour), which per-PR CI has never run.

The shared .github/actions/contract-tests composite gains an enable_long_running_tests input (default false). When set to true, both the standard and FDv2 harness legs append -enable-long-running-tests to extra_params, so existing sdk-server-ci / sdk-client-ci jobs stay unchanged.

sdk-server-nightly.yml runs daily at 04:00 UTC (plus workflow_dispatch with optional branch), uses a 120-minute job timeout, invokes the composite with run_fdv2_tests and enable_long_running_tests enabled, and always uploads test-service.log.

Reviewed by Cursor Bugbot for commit ac8a365. Bugbot is set up for automated code reviews on this repo. Configure here.

The test harness gates its RETRY-conformance and FDv2 long-running tests
behind -enable-long-running-tests. Per-PR CI does not pass that flag, so
none of them have ever run for this SDK, even though the contract test
service declares both retry-conformance-fdv1-streaming and
retry-conformance-fdv1-polling.

Adds a nightly workflow that enables them, plus an enable_long_running_tests
input on the contract-tests action to carry the flag. The input defaults to
false, so per-PR CI for the server and client SDKs renders identical params
to today.

The job is expected to fail on three FDv2 tests until those are addressed
separately:
  streaming/fdv2/recoverable fallback to secondary synchronizer
  streaming/fdv2/recoverable fallback with recovery
  streaming/fdv2/permanent fallback with recovery
@tanderson-ld
tanderson-ld requested a review from a team as a code owner October 23, 2026 14:19

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant