Repository navigation
ci: make CI pass by testing what EMQX ships - #16
Merged
Merged
Conversation
The test dependencies floated on master: gun, ct_helper, ci.erlang.mk and quicer (main). erlang.mk resolves each dependency once, so the cowboy server and the gun client in a test run share one cowlib. With gun master, that cowlib was a version neither was developed against. Tests failed in cowlib parsing and serialising code, for example gun raising invalid_request_line and the HTTP/2 push and 101 cases. EMQX builds this branch with cowlib 2.14.0 and gun 2.1.0. Pin to those so CI tests what ships: - cowlib 2.15.0 -> 2.14.0 (also in rebar.config) - gun master -> 2.1.0 - ct_helper master -> 9df44e9 - ci.erlang.mk master -> 1497ed9 - quicer main -> 0.4.11, the version EMQX uses
The h3 and h3_compress groups and the rfc9114, rfc9204 and rfc9220 suites cannot run here. gun 2.1.0, the client version EMQX ships, has no HTTP/3 support, and CI does not build quicer. Every run failed in init_per_group with a missing quicer NIF. Upstream disabled the same groups in 90c7a3e ("Temporarily disable HTTP/3 tests"). This commit comments them out the same way, so a later upstream sync applies cleanly.
(cherry picked from commit e524162)
Neither function returns: both end in terminate/3 or in a loop that
exits. Dialyzer on OTP 28 reports both as having no local return. The
specs record that intent and clear the warnings. Behaviour does not
change.
The reason argument is any(), not terminate_reason(). Callers pass
{error, {sock_error, Reason}}, which terminate_reason() does not cover,
and Dialyzer rejects the narrower spec.
The workflow called ninenines/ci.erlang.mk's reusable workflow at master. That workflow builds a matrix of OTP 24 to 29 and master, on Ubuntu, macOS and Windows. It also checks out ci.erlang.mk at master inside each job, so the matrix and the CI logic changed without a commit here. It sets COWBOY_QUICER=0, so HTTP/3 always failed. EMQX builds this branch into 5.10 (OTP 27, emqx/ranch 1.8.1-emqx-1), 6.0 (OTP 27, emqx/ranch 2.2.0-emqx-3) and 6.1 and later (OTP 28, emqx/ranch 2.2.0-emqx-3), all on Linux. Replace the reusable workflow with a pinned one that tests these three combinations. Dialyzer runs on OTP 28 with COWBOY_QUICER=1, which compiles quicer's Erlang modules so cowboy_http3 can be checked. Dropped: OTP 24, 25, 26, 29 and master, macOS, Windows, and the examples job. The examples are not part of what EMQX ships. Push CI now also runs on emqx-2.13.x, the branch this fork works on, so a merge that breaks CI shows up there. The weekly schedule and its cache cleanup job are removed. With pinned versions there is nothing new for them to test.
(cherry picked from commit cbe5c53)
zmstone
marked this pull request as ready for review
October 5, 2026 09:20
qzhuyan
approved these changes
Oct 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI on this fork has not passed in its last 60 runs. This PR makes it pass by testing the dependency and OTP versions that EMQX ships.
Diagnosis: dependency skew
The fork's CI tested a dependency combination that no release uses.
dev-510)dev-60)dev-61tomaster)master(2.6.0+)erlang.mk resolves each dependency once. The cowboy server and the gun client in a test run therefore share one cowlib, and with gun
masterneither side was developed against that version. Failures landed in cowlib parsing and serialising code:{invalid_request_line, path, ...}instatic_handler_SUITE;req_SUITE:do_push_http2_origin,rfc7231_SUITE:status_code_101andstream_handler_SUITE:do_terminate_on_switch_protocolfailed on response mismatches.I reproduced all four failures locally on OTP 27 before changing anything. With the pins below, all four pass and the code under test is unchanged.
Pins
rebar.config)mastermaster9df44e9master1497ed9maincowlib moves back to 2.14.0. 2.15.0 was a test-only bump (
8806d2e) to follow gunmaster, and EMQX overrides cowlib to 2.14.0 in every release that uses this branch.The Makefile default for ranch stays at upstream 1.8.1. CI overrides it with
dep_ranch=...for each matrix entry, so the generatedrebar.configdoes not change for other consumers.Workflow
The workflow called
ninenines/ci.erlang.mk/.github/workflows/ci.yaml@master. That workflow also checks out ci.erlang.mk atmasterinside each job, so pinning only theuses:SHA would not have pinned the CI logic. It setsCOWBOY_QUICER=0when no repo variable is set, so HTTP/3 always failed. This PR replaces it with a workflow in this repo.Kept:
ubuntu-24.04.COWBOY_QUICER=1. That setting compiles quicer's Erlang modules socowboy_http3is analysed. The Makefile already skips quicer's NIF build.Dropped:
examplesjob. The examples are not part of what EMQX ships.Added
emqx-2.13.xto the push trigger. Until now push CI ran only onmaster, so this branch had no baseline run.Dialyzer
Dialyzer has never passed in this fork's CI. On OTP 29 the build failed on a new compiler warning before Dialyzer ran. On OTP 28 with the pinned versions it reported:
cowboy_http2"invalid type specification" warnings. Upstream 2.13.0 has the same warnings, and upstream fixed them ine524162("Fix cowboy_http2 Dialyzer warnings with OTP-28"). This PR cherry-picks that commit with-x.cowboy_websocket_linger:websocket_closed/3,4has no local return. Both functions are meant not to return. This PR addsno_return()specs. Behaviour does not change. The reason argument is specced asany(), because callers pass{error, {sock_error, _}}and the module'sterminate_reason()type does not cover that tuple.Flaky test fix
The first CI run of this PR failed
http_SUITE:graceful_shutdown_listeneron OTP 28. The test let a 500 ms handler race the listener shutdown, and the response sometimes went out withoutconnection: close. Upstream fixed this flake incbe5c53("Resolve http_SUITE:graceful_shutdown_listener flake"). This PR cherry-picks that commit with-x. The test is now ordered by a trace on the connection process and does not depend on timing.Skipped tests
h3andh3_compressgroups incowboy_test:common_all/0andcompress_SUITE:all/0, and therfc9114,rfc9204andrfc9220suites. gun 2.1.0 has no HTTP/3 client, and CI does not build quicer's NIF. Upstream disabled the same groups in90c7a3e. This PR comments them out the same way, so a later upstream sync applies cleanly. Each skip has a comment that states the reason.No other test is skipped or changed. The websocket linger suites and the #15 regression test run on every matrix entry.
Local environment notes
These failed locally only. CI does not hit them:
rest_handler_SUITErange cases andstatic_handler_SUITE:bad_priv_*:code:lib_dir(cowboy)returns{error, bad_name}when the checkout directory is not namedcowboy. CI checks out intocowboy/.sys_SUITE*_loopcases:long_polling_sys_hreplies after a 500 ms timer, and the test must act before it fires. They failed only while the host was loaded with parallel jobs.proxy_header_SUITE:fail_gracefully_on_disconnect: failed once withnoprocunder the same load. The server process can exit before the test monitors it.Verification
make -k testsexit 0.make -k testsexit 0.COWBOY_QUICER=1 make dialyzeexit 0.