Repository navigation
ci: add nightly contract tests with long-running tests enabled - #380
Open
tanderson-ld wants to merge 1 commit into
Open
tanderson-ld wants to merge 1 commit into
tanderson-ld wants to merge 1 commit into
Conversation
The test harness gates its RETRY-conformance and FDv2 long-running tests behind -enable-long-running-tests. Per-PR CI does not pass that flag, so none of them have ever run for this SDK, even though the contract test service declares both retry-conformance-fdv1-streaming and retry-conformance-fdv1-polling. Adds a nightly workflow that enables them, plus an enable_long_running_tests input on the contract-tests action to carry the flag. The input defaults to false, so per-PR CI for the server and client SDKs renders identical params to today. The job is expected to fail on three FDv2 tests until those are addressed separately: streaming/fdv2/recoverable fallback to secondary synchronizer streaming/fdv2/recoverable fallback with recovery streaming/fdv2/permanent fallback with recovery
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requirements
Related issues
Tracked in Jira under the RETRY-conformance epic. The issue key is omitted from this body and commit deliberately: the ticket also requires the FDv2 failures below to be tracked, and that follow-up ticket does not exist yet, so an auto-close on merge would be premature.
Counterparts already merged: java-core PR 212 and go-server-sdk PR 454.
Describe the solution you've provided
The harness gates its RETRY-conformance and FDv2 long-running tests behind
-enable-long-running-tests, because they wait on real backoff timing and take about an hour. Per-PR CI does not pass it, so none of them have ever run for this SDK — even though the contract test service has declaredretry-conformance-fdv1-streamingandretry-conformance-fdv1-pollingsince the RETRY work landed.Two changes:
.github/actions/contract-tests/action.yml— newenable_long_running_testsinput, appended to the params of both harness legs. It defaults to'false', sosdk-server-ci.ymlandsdk-client-ci.ymlrender byte-identical params to today..github/workflows/sdk-server-nightly.yml— new.scheduleat 04:00 UTC plusworkflow_dispatch, single runner,timeout-minutes: 120, test service log uploaded onalways().Warning
This job is expected to fail on three tests, all on the FDv2 leg:
These are deliberately not suppressed. A nightly that goes red on a real defect is the point — that is how the equivalent Go defect was found and fixed. More than three failures means something else broke.
Worth knowing: these are the same three tests java-core suppresses in
test-supressions-fdv2.txt, labelled there as "pending spec alignment between Go and Java." That label looks wrong — go-server-sdk passes all three, so this is a shared Java/.NET implementation gap rather than a spec disagreement. A follow-up ticket covering both SDKs is pending.Describe alternatives you've considered
-junitis a harness flag unrelated to the .NET toolchain, GitHub Actions does not parse an uploaded JUnit file without a separate reporter action, and the harness already prints failing test names to the job log.Additional context
Verified locally against the real service (
dotnet-server-sdk 8.17.2.0), replaying the exact params the composite renders:use -enable-long-running-tests to runskip lines on either leg, confirming the flag reaches both.So the RETRY-conformance work in this SDK passes in full; the failures are confined to FDv2 synchronizer fallback and recovery.
Note
Overview
Adds a nightly Server SDK contract-test workflow and a way to turn on harness tests that wait on real retry backoff (~1 hour), which per-PR CI has never run.
The shared
.github/actions/contract-testscomposite gains anenable_long_running_testsinput (defaultfalse). When set totrue, both the standard and FDv2 harness legs append-enable-long-running-teststoextra_params, so existingsdk-server-ci/sdk-client-cijobs stay unchanged.sdk-server-nightly.ymlruns daily at 04:00 UTC (plusworkflow_dispatchwith optional branch), uses a 120-minute job timeout, invokes the composite withrun_fdv2_testsandenable_long_running_testsenabled, and always uploadstest-service.log.Reviewed by Cursor Bugbot for commit ac8a365. Bugbot is set up for automated code reviews on this repo. Configure here.