Skip to content

ci: make CI pass by testing what EMQX ships - #16

Merged
zmstone merged 6 commits into
emqx-2.13.xfrom
261002-fix-ci
Oct 5, 2026
Merged

zmstone merged 6 commits into
emqx-2.13.xfrom
261002-fix-ci

Conversation

@zmstone

@zmstone zmstone commented Oct 2, 2026 •

Copy link
Copy Markdown
Member

CI on this fork has not passed in its last 60 runs. This PR makes it pass by testing the dependency and OTP versions that EMQX ships.

Diagnosis: dependency skew

The fork's CI tested a dependency combination that no release uses.

cowlib gun ranch OTP
EMQX 5.10 (dev-510) 2.14.0 2.1.0 emqx/ranch 1.8.1-emqx-1 27
EMQX 6.0 (dev-60) 2.14.0 2.1.0 emqx/ranch 2.2.0-emqx-3 27
EMQX 6.1+ (dev-61 to master) 2.14.0 2.1.0 emqx/ranch 2.2.0-emqx-3 28
this fork's CI before this PR 2.15.0 master (2.6.0+) ranch 1.8.1 24 to 29, master

erlang.mk resolves each dependency once. The cowboy server and the gun client in a test run therefore share one cowlib, and with gun master neither side was developed against that version. Failures landed in cowlib parsing and serialising code:

  • gun raised {invalid_request_line, path, ...} in static_handler_SUITE;
  • req_SUITE:do_push_http2_origin, rfc7231_SUITE:status_code_101 and stream_handler_SUITE:do_terminate_on_switch_protocol failed on response mismatches.

I reproduced all four failures locally on OTP 27 before changing anything. With the pins below, all four pass and the code under test is unchanged.

Pins

Dependency Before After
cowlib 2.15.0 2.14.0 (Makefile and rebar.config)
gun master 2.1.0
ct_helper master 9df44e9
ci.erlang.mk master 1497ed9
quicer main 0.4.11, the version EMQX uses
ranch (CI only) 1.8.1 per matrix entry, see below

cowlib moves back to 2.14.0. 2.15.0 was a test-only bump (8806d2e) to follow gun master, and EMQX overrides cowlib to 2.14.0 in every release that uses this branch.

The Makefile default for ranch stays at upstream 1.8.1. CI overrides it with dep_ranch=... for each matrix entry, so the generated rebar.config does not change for other consumers.

Workflow

The workflow called ninenines/ci.erlang.mk/.github/workflows/ci.yaml@master. That workflow also checks out ci.erlang.mk at master inside each job, so pinning only the uses: SHA would not have pinned the CI logic. It sets COWBOY_QUICER=0 when no repo variable is set, so HTTP/3 always failed. This PR replaces it with a workflow in this repo.

Kept:

  • Tests on three OTP and ranch combinations: OTP 27.3.4.18 with emqx/ranch 1.8.1-emqx-1 (5.10), OTP 27.3.4.18 with emqx/ranch 2.2.0-emqx-3 (6.0), and OTP 28.5.0.7 with emqx/ranch 2.2.0-emqx-3 (6.1 and later). All on ubuntu-24.04.
  • Dialyzer on OTP 28 with emqx/ranch 2.2.0-emqx-3 and COWBOY_QUICER=1. That setting compiles quicer's Erlang modules so cowboy_http3 is analysed. The Makefile already skips quicer's NIF build.

Dropped:

  • OTP 24, 25, 26, 29 and master. Only EMQX 5.10 and later use this branch, and those run OTP 27 and 28. EMQX 5.8 and 5.9 use cowboy 2.9.2.
  • macOS and Windows. EMQX does not run this branch on them in production.
  • The examples job. The examples are not part of what EMQX ships.
  • The weekly schedule and its cache cleanup job. With every version pinned, a scheduled run has nothing new to test.

Added emqx-2.13.x to the push trigger. Until now push CI ran only on master, so this branch had no baseline run.

Dialyzer

Dialyzer has never passed in this fork's CI. On OTP 29 the build failed on a new compiler warning before Dialyzer ran. On OTP 28 with the pinned versions it reported:

  • five cowboy_http2 "invalid type specification" warnings. Upstream 2.13.0 has the same warnings, and upstream fixed them in e524162 ("Fix cowboy_http2 Dialyzer warnings with OTP-28"). This PR cherry-picks that commit with -x.
  • two warnings in the fork's own code: cowboy_websocket_linger:websocket_closed/3,4 has no local return. Both functions are meant not to return. This PR adds no_return() specs. Behaviour does not change. The reason argument is specced as any(), because callers pass {error, {sock_error, _}} and the module's terminate_reason() type does not cover that tuple.

Flaky test fix

The first CI run of this PR failed http_SUITE:graceful_shutdown_listener on OTP 28. The test let a 500 ms handler race the listener shutdown, and the response sometimes went out without connection: close. Upstream fixed this flake in cbe5c53 ("Resolve http_SUITE:graceful_shutdown_listener flake"). This PR cherry-picks that commit with -x. The test is now ordered by a trace on the connection process and does not depend on timing.

Skipped tests

  • HTTP/3: the h3 and h3_compress groups in cowboy_test:common_all/0 and compress_SUITE:all/0, and the rfc9114, rfc9204 and rfc9220 suites. gun 2.1.0 has no HTTP/3 client, and CI does not build quicer's NIF. Upstream disabled the same groups in 90c7a3e. This PR comments them out the same way, so a later upstream sync applies cleanly. Each skip has a comment that states the reason.

No other test is skipped or changed. The websocket linger suites and the #15 regression test run on every matrix entry.

Local environment notes

These failed locally only. CI does not hit them:

  • rest_handler_SUITE range cases and static_handler_SUITE:bad_priv_*: code:lib_dir(cowboy) returns {error, bad_name} when the checkout directory is not named cowboy. CI checks out into cowboy/.
  • sys_SUITE *_loop cases: long_polling_sys_h replies after a 500 ms timer, and the test must act before it fires. They failed only while the host was loaded with parallel jobs.
  • proxy_header_SUITE:fail_gracefully_on_disconnect: failed once with noproc under the same load. The server process can exit before the test monitors it.

Verification

  • Local, OTP 27 with emqx/ranch 1.8.1-emqx-1: make -k tests exit 0.
  • Local, OTP 28 with emqx/ranch 2.2.0-emqx-3: make -k tests exit 0.
  • Local, OTP 28 with emqx/ranch 2.2.0-emqx-3: COWBOY_QUICER=1 make dialyze exit 0.
  • First green CI run: https://github.com/emqx/cowboy/actions/runs/37007188215

zmstone and others added 6 commits October 2, 2026 13:56
The test dependencies floated on master: gun, ct_helper, ci.erlang.mk
and quicer (main). erlang.mk resolves each dependency once, so the
cowboy server and the gun client in a test run share one cowlib. With
gun master, that cowlib was a version neither was developed against.
Tests failed in cowlib parsing and serialising code, for example gun
raising invalid_request_line and the HTTP/2 push and 101 cases.

EMQX builds this branch with cowlib 2.14.0 and gun 2.1.0. Pin to those
so CI tests what ships:

- cowlib 2.15.0 -> 2.14.0 (also in rebar.config)
- gun master -> 2.1.0
- ct_helper master -> 9df44e9
- ci.erlang.mk master -> 1497ed9
- quicer main -> 0.4.11, the version EMQX uses
The h3 and h3_compress groups and the rfc9114, rfc9204 and rfc9220
suites cannot run here. gun 2.1.0, the client version EMQX ships, has
no HTTP/3 support, and CI does not build quicer. Every run failed in
init_per_group with a missing quicer NIF.

Upstream disabled the same groups in 90c7a3e ("Temporarily disable
HTTP/3 tests"). This commit comments them out the same way, so a later
upstream sync applies cleanly.
Neither function returns: both end in terminate/3 or in a loop that
exits. Dialyzer on OTP 28 reports both as having no local return. The
specs record that intent and clear the warnings. Behaviour does not
change.

The reason argument is any(), not terminate_reason(). Callers pass
{error, {sock_error, Reason}}, which terminate_reason() does not cover,
and Dialyzer rejects the narrower spec.
The workflow called ninenines/ci.erlang.mk's reusable workflow at
master. That workflow builds a matrix of OTP 24 to 29 and master, on
Ubuntu, macOS and Windows. It also checks out ci.erlang.mk at master
inside each job, so the matrix and the CI logic changed without a
commit here. It sets COWBOY_QUICER=0, so HTTP/3 always failed.

EMQX builds this branch into 5.10 (OTP 27, emqx/ranch 1.8.1-emqx-1),
6.0 (OTP 27, emqx/ranch 2.2.0-emqx-3) and 6.1 and later (OTP 28,
emqx/ranch 2.2.0-emqx-3), all on Linux. Replace the reusable workflow
with a pinned one that tests these three combinations. Dialyzer runs on
OTP 28 with COWBOY_QUICER=1, which compiles quicer's Erlang modules so
cowboy_http3 can be checked.

Dropped: OTP 24, 25, 26, 29 and master, macOS, Windows, and the
examples job. The examples are not part of what EMQX ships.

Push CI now also runs on emqx-2.13.x, the branch this fork works on, so
a merge that breaks CI shows up there. The weekly schedule and its
cache cleanup job are removed. With pinned versions there is nothing
new for them to test.
@zmstone
zmstone marked this pull request as ready for review October 5, 2026 09:20
@zmstone
zmstone merged commit 9014626 into emqx-2.13.x Oct 5, 2026
4 checks passed
@zmstone
zmstone deleted the 261002-fix-ci branch October 5, 2026 10:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants