Skip to content

Add Matrix/Synapse postoverflow whitelist - #1726

Open
pscriptos wants to merge 5 commits into
crowdsecurity:masterfrom
pscriptos:matrix-whitelist
Open

pscriptos wants to merge 5 commits into
crowdsecurity:masterfrom
pscriptos:matrix-whitelist

Conversation

@pscriptos

@pscriptos pscriptos commented Mar 13, 2026 •

Copy link
Copy Markdown

Description

Postoverflow whitelist for known false positives caused by legitimate Matrix/Synapse traffic. It replaces the original parser whitelist from this PR, which was too broad.

An overflow is only discarded when it comes from one of the scenarios below and all of its events are known harmless requests (method, status and exact path). Any other request in the same overflow (e.g. GET /_matrix/.env) still raises the alert, and all other scenarios keep seeing Matrix traffic.

Scenario Requests Reason
crowdsecurity/http-probing GET → 404 on /_matrix/client/(r0|v3)/rooms/{roomId}/state/m.room.encryption clients checking unencrypted rooms (spec behaviour)
crowdsecurity/http-probing GET → 404 on /_matrix/federation/v1/event/{eventId} remote homeservers fetching unknown events
crowdsecurity/http-crawl-non_statics GET → 200/304 on /_matrix/client/v1/media/thumbnail/{server}/{mediaId} clients loading avatars/previews in bursts (no file extension, so not detected as static)

Not whitelisted on purpose: /login, /_synapse/ (admin API), non-GET requests and 403 responses.

Note: with Traefik, http_path includes the query string, so the path regexes allow an optional ?... suffix.

Checklist

  • I have read the contributing guide
  • I have tested my changes locally
  • For new parsers or scenarios, tests have been added (.tests/matrix-whitelist, .tests/matrix-whitelist-crawl)
  • I have run the hub linter and no issues were reported (see contributing guide)
  • Automated tests are passing (cscli hubtest run locally with CrowdSec v1.8.2)
  • AI was used to generate any/all content of this PR

@blotus

blotus commented Mar 18, 2026

Copy link
Copy Markdown
Member

Hello,

Thanks for the PR.

Unfortunately, I think using a parser whitelist for that is a bit too broad: you are effectively ignoring every request on anything that starts with those paths.

A much better solution would be to use a postoverflow to only allow those paths for specific scenarios.

@pscriptos

Copy link
Copy Markdown
Author

Thank you very much for your feedback.
I’ll look into it shortly and get back to you.
Have a good weekend.

@Nemric

Nemric commented Mar 22, 2026 •

Copy link
Copy Markdown

Hi, I'm facing the same issue, then came here, I did follow the link about postoverflow, here is a file I'm now trying

name: synapse-client-whitelists
description: Whitelist synapse client application for crawl non static
whitelist:
  reason: synapse-client can trigger FP
  expression:
    - evt.Overflow.Alert.Scenario == "crowdsecurity/http-crawl-non_statics" and all(evt.Overflow.Alert.Events, {.GetMeta("http_path") startsWith "/_matrix/client/"})
    - evt.Overflow.Alert.Scenario == "crowdsecurity/http-probing" and all(evt.Overflow.Alert.Events, {.GetMeta("http_path") startsWith "/_matrix/client/"})

I did add the http-probing scenario because in some rooms there are a lot of 404 😕 that trigger the ban
I did just restart crowdsec, so, I don't yet now if it works as great as expected ^^

[edit]
After a couple of days, it looks like it works !
(the path I had this is : ./postoverflows/s01-whitelist/synapse-client.yaml)

@gemarcano

Copy link
Copy Markdown

The suggested postoverflow didn't work for me. Part of it is that some 404s caused by Nextcloud activity seem to be causing the bucket to overflow and then everything goes sideways.

Based on what I'm seeing, a parser whitelist may be the better option, if there are other services that may also be generating 404s... Nextcloud is notorious for being bad about 404s... even with https://app.crowdsec.net/hub/author/crowdsecurity/configurations/nextcloud-whitelist .

@Nemric

Nemric commented Mar 30, 2026

Copy link
Copy Markdown

The suggested postoverflow didn't work for me. Part of it is that some 404s caused by Nextcloud activity seem to be causing the bucket to overflow and then everything goes sideways.

Based on what I'm seeing, a parser whitelist may be the better option, if there are other services that may also be generating 404s... Nextcloud is notorious for being bad about 404s... even with https://app.crowdsec.net/hub/author/crowdsecurity/configurations/nextcloud-whitelist .

this PR is the whitelist about matrix/synapse

Copilot AI review requested due to automatic review settings March 30, 2026 11:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new enrich-stage HTTP whitelist parser intended to prevent legitimate Matrix/Synapse endpoints from being counted toward generic HTTP scanning/crawling scenarios (e.g., http-probing, http-crawl-non_statics).

Changes:

  • Introduces a Matrix/Synapse path-prefix whitelist parser (/_matrix/, /_synapse/, /.well-known/matrix/).
  • Adds documentation describing why these endpoints commonly generate benign “suspicious-looking” HTTP patterns.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File Description
parsers/s02-enrich/crowdsecurity/matrix-whitelist.yaml Adds a whitelist parser matching Matrix/Synapse request paths.
parsers/s02-enrich/crowdsecurity/matrix-whitelist.md Documents the rationale and covered Matrix/Synapse endpoints.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@@ -0,0 +1,9 @@
name: my/matrix-whitelist

Copilot AI Mar 30, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

name is set to my/matrix-whitelist, but all other parsers in parsers/s02-enrich/crowdsecurity/ use the crowdsecurity/ namespace (e.g. parsers/s02-enrich/crowdsecurity/actual-budget-whitelist.yaml:1). Using a different namespace is likely to fail hub conventions/linting and makes the parser harder to discover; rename it to crowdsecurity/matrix-whitelist (and keep it consistent with the file path).

Suggested change
name: my/matrix-whitelist
name: crowdsecurity/matrix-whitelist

Copilot uses AI. Check for mistakes.
@@ -0,0 +1,9 @@
name: my/matrix-whitelist
description: "Whitelist Matrix/Synapse requests from NPMplus logs"

Copilot AI Mar 30, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The description says "from NPMplus logs", but the filter matches any event with evt.Meta.service == 'http' and the standard http_access-log/http_error-log log types. Either scope the filter to NPMplus-specific events (if that’s the intent) or update the description to reflect that this whitelist applies to generic HTTP logs.

Suggested change
description: "Whitelist Matrix/Synapse requests from NPMplus logs"
description: "Whitelist Matrix/Synapse HTTP requests from generic http_access-log/http_error-log entries"

Copilot uses AI. Check for mistakes.
Comment on lines +7 to +9
- "evt.Meta.http_path startsWith '/_matrix/'"
- "evt.Meta.http_path startsWith '/_synapse/'"
- "evt.Meta.http_path startsWith '/.well-known/matrix/'"

Copilot AI Mar 30, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This whitelists all requests to /_matrix/ and /_synapse/ purely by path prefix. Unlike other HTTP app whitelists in this directory (which typically narrow by verb/status and specific endpoints), this creates a broad blind spot where probing/bruteforce against Synapse/Matrix endpoints would no longer contribute to remediation decisions. Consider narrowing the whitelist conditions (e.g., to specific verbs/statuses/endpoints known to FP) or moving the exception into the specific scenarios that FP (so Matrix-specific attacks can still be detected).

Suggested change
- "evt.Meta.http_path startsWith '/_matrix/'"
- "evt.Meta.http_path startsWith '/_synapse/'"
- "evt.Meta.http_path startsWith '/.well-known/matrix/'"
- "evt.Meta.http_verb == 'GET' && evt.Meta.http_path startsWith '/.well-known/matrix/'"

Copilot uses AI. Check for mistakes.
Comment on lines +1 to +9
name: my/matrix-whitelist
description: "Whitelist Matrix/Synapse requests from NPMplus logs"
filter: "evt.Meta.service == 'http' && evt.Meta.log_type in ['http_access-log', 'http_error-log']"
whitelist:
reason: "Matrix federation/client traffic"
expression:
- "evt.Meta.http_path startsWith '/_matrix/'"
- "evt.Meta.http_path startsWith '/_synapse/'"
- "evt.Meta.http_path startsWith '/.well-known/matrix/'"

Copilot AI Mar 30, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A new whitelist parser is added, but there is no corresponding .tests/matrix-whitelist/ (most existing whitelists here have dedicated tests, e.g. .tests/actual-budget-whitelist/, .tests/calibre-web-whitelist/, .tests/nextcloud-whitelist/). Add a minimal test fixture asserting that Matrix/Synapse paths are marked whitelisted and that non-Matrix paths are not.

Copilot uses AI. Check for mistakes.
@gemarcano

Copy link
Copy Markdown

this PR is the whitelist about matrix/synapse

Sorry, I wasn't clear enough. The combination of matrix/synapse and Nextcloud endpoints 404'ing made the postoverflow whitelist to fail. Or really, if at any point any other 404 happens during that window, that would make the postoverflow whitelist to fail thanks to all() if I'm not misinterpreting that. I tried using any, but it was somehow also not quite working right... but that wouldn't be the right approach either, right?

The contents of this PR do seem to work for me, even if allows for a lot of request URLs.

@miguelrdev

Copy link
Copy Markdown

I'm also using Synapse and getting banned due to the scenario crowdsecurity/http-probing.

@sabban sabban added the waiting-for-user Waiting for submitter answer regarding specific points in the PR label May 12, 2026
@liamthexpl0rer

Copy link
Copy Markdown

@pscriptos Is there any progress? I have the same problem and it would be very nice if this can be merged soon.

@liamthexpl0rer

Copy link
Copy Markdown

Hi, I'm facing the same issue, then came here, I did follow the link about postoverflow, here is a file I'm now trying

name: synapse-client-whitelists
description: Whitelist synapse client application for crawl non static
whitelist:
  reason: synapse-client can trigger FP
  expression:
    - evt.Overflow.Alert.Scenario == "crowdsecurity/http-crawl-non_statics" and all(evt.Overflow.Alert.Events, {.GetMeta("http_path") startsWith "/_matrix/client/"})
    - evt.Overflow.Alert.Scenario == "crowdsecurity/http-probing" and all(evt.Overflow.Alert.Events, {.GetMeta("http_path") startsWith "/_matrix/client/"})

I did add the http-probing scenario because in some rooms there are a lot of 404 😕 that trigger the ban I did just restart crowdsec, so, I don't yet now if it works as great as expected ^^

[edit] After a couple of days, it looks like it works ! (the path I had this is : ./postoverflows/s01-whitelist/synapse-client.yaml)

Yesterday I applied this rule to my system and it works too. If @pscriptos does not react, can you maybe open an new PR @Nemric?

@sabban

sabban commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

Hi,

This PR still needs some attention. And some whitelists seem a bit broader than mandatory. I think this should be done by someone using matrix/synapse.

regards,

@liamthexpl0rer

liamthexpl0rer commented Oct 30, 2026 •

Copy link
Copy Markdown

Hi,

This PR still needs some attention. And some whitelists seem a bit broader than mandatory. I think this should be done by someone using matrix/synapse.

regards,

I'm open to investigate if someone can tell me what I should test. You can see my current NixOS configuration at Codeberg (links below). ;)

@sabban Do you have an idea for a rule that I can test?

https://codeberg.org/liamthexpl0rer/nixos-serverconf/src/commit/d9368e510b9945958501927e1e581e1921ce4c68/modules/apps/security/crowdsec.nix

https://codeberg.org/liamthexpl0rer/nixos-serverconf/src/commit/d9368e510b9945958501927e1e581e1921ce4c68/modules/apps/social/matrix-synapse.nix#L248

@liamthexpl0rer

Copy link
Copy Markdown

I also found the official API specifications for the matrix protocol.
https://spec.matrix.org/v1.19/client-server-api/

@pscriptos

Copy link
Copy Markdown
Author

Hi everyone, sorry for the long silence on this one, and thanks for all the feedback and testing.

I've picked this up again and agree with @blotus and @sabban: the parser whitelist is too broad. It drops every event under /_matrix/ and /_synapse/ before any scenario sees it, so probes like /_matrix/.env or brute force against /_matrix/client/v3/login would go unnoticed. On my own setup (Synapse behind Traefik) it was silently swallowing about 50% of all HTTP events.

Looking at my logs, the actual false positives come from a very small set of legitimate 404s:

  • GET /_matrix/client/v3/rooms/{roomId}/state/m.room.encryption/: clients checking unencrypted rooms
  • GET /_matrix/federation/v1/event/{eventId}: remote homeservers fetching events

Because every room or event ID is a distinct path, these trip http-probing (and occasionally http-crawl-non_statics).

What I've changed locally and am currently testing:

  • Replaced the parser whitelist with a postoverflow (as suggested), scoped to http-probing and http-crawl-non_statics only
  • Only applies if all events of the overflow are GET/HEAD requests to the Matrix client/federation/media/key APIs
  • /login and /_synapse/ (admin API) are explicitly not whitelisted
  • All other scenarios still see Matrix traffic normally

First results look good: within the first minutes it already prevented several bans of federating homeservers. I'll let it run for a few days, then update this PR accordingly (correct namespace, description, hub tests under .tests/).

Thanks for your patience!

@sabban

sabban commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Hi @pscriptos, thanks for picking this up again, and thanks @Nemric, @liamthexpl0rer and @gemarcano for testing.

Your plan (a postoverflow scoped to the scenarios that false-positive) is the right direction, as @blotus suggested. To keep it as narrow as possible:

  • put it in postoverflows/s01-whitelist/whatever-fits-you/matrix-whitelist.yaml, named whatever-fits-you/matrix-whitelist
  • scope each rule to one scenario, and only whitelist an overflow when all its events are known-harmless Matrix requests: method, status and an exact path pattern, not a /_matrix/ prefix
  • start with the two false positives you found, and add more only when backed by a real log line. Keep /login, /_synapse/ and non-GET requests out.

Something like this works in a quick hubtest on my side:

name: whatever-fits-you/matrix-whitelist
description: "Whitelist known false positives from Matrix/Synapse clients and federation"
whitelist:
  reason: "Matrix/Synapse known false positives"
  expression:
    - |
      evt.Overflow.Alert.GetScenario() == 'crowdsecurity/http-probing' && all(evt.Overflow.Alert.Events, {
        .GetMeta('http_verb') == 'GET' && .GetMeta('http_status') == '404' && (
          .GetMeta('http_path') matches '^/_matrix/client/(r0|v3)/rooms/[^/]+/state/m\\.room\\.encryption/?$' ||
          .GetMeta('http_path') matches '^/_matrix/federation/v1/event/[^/]+$'
        )
      })

If http-crawl-non_statics also false-positives, please add a separate rule for it, based on the real requests and status codes you see (they're likely 200, not 404).

For tests, add .tests/matrix-whitelist/ with crowdsecurity/http-probing as the scenario and this file under postoverflows:. Cover both cases: a burst of these Matrix 404s must give results[N].Overflow.Whitelisted == true, and the same burst plus a real probe (e.g. GET /_matrix/.env) must still raise an alert.

About @gemarcano's point: with all(), an overflow mixing Matrix 404s with other 404s is not whitelisted, and that's intended. Those other 404s should be handled by their own app whitelist (e.g. crowdsecurity/nextcloud-whitelist), which removes them before they reach the bucket.

@liamthexpl0rer, if you can run this against your own logs and report which false positives remain, that would help a lot.

Regards,

@plule

plule commented Nov 5, 2026

Copy link
Copy Markdown

Hello, I’m also trying to build filters for matrix/synapse, here a some cases that I don’t think are covered by the more "restricted" rules:

Synapse’s media serving is not detected as "static" by the native http-logs, thus easily triggering the http-crawl-non_statics scenario. They are however all served under /_matrix/client/v1/media/. I have no experience in crowdsecurity, so this might not be ideal, but patching evt.Parsed.static_ressource works well in my deployment:

filter: "evt.Meta.service == 'http' && evt.Meta.log_type in ['http_access-log', 'http_error-log'] && evt.Meta.http_path startsWith '/_matrix/client/v1/media/'"
statics:
- target: evt.Parsed.static_ressource
  value: 'true'

With regards to the http-probing scenario, other paths to consider, that I saw 404’d legitimately:

  • /_matrix/client/v1/auth_metadata: Expected for servers without OAuth
  • /_matrix/client/unstable/org.matrix.msc3814.v1/dehydrated_device: I think that clients are expected to probe unstable API and 404
  • many thumbnails from /_matrix/client/v1/media/thumbnail/<server_url>/*: Possibly as a result of a media disk crash a few years ago

@liamthexpl0rer

This comment was marked as outdated.

@liamthexpl0rer

Copy link
Copy Markdown

I've found some other false-positives and extended the whitelist. I will wait one week if there are still any alerts and post the whitelist then. ^^

I replaced the broad parser whitelist with a postoverflow that only
discards http-probing and http-crawl-non_statics overflows when all
events are known harmless Matrix/Synapse requests (method, status and
exact path), and added hub tests for both scenarios.
@pscriptos pscriptos changed the title Add Matrix/Synapse parser whitelist and description Add Matrix/Synapse postoverflow whitelist Nov 11, 2026
@pscriptos

Copy link
Copy Markdown
Author

Thanks a lot @sabban for the detailed guidance, and @plule and @liamthexpl0rer for the additional data.

I've updated the PR accordingly:

  • Replaced the parser whitelist with postoverflows/s01-whitelist/pscriptos/matrix-whitelist.yaml
  • One rule per scenario, only matching when all events are known harmless (method + status + exact path)
  • http-probing: your rule, with one change: Traefik's http_path meta includes the query string, so the regexes now allow an optional ?... suffix (otherwise the $ anchor never matches requests with parameters)
  • http-crawl-non_statics: separate rule for GET 200/304 on /_matrix/client/v1/media/thumbnail/..., as @plule reported
  • Hub tests for both scenarios: a pure Matrix burst gets Whitelisted == true, the same burst plus a real probe (/_matrix/.env) or a crawl of other paths still raises an alert

Evidence from my own setup (Synapse behind Traefik, ~4 days of logs):

  • http-crawl-non_statics would have fired 11 times, all on media thumbnails from my own clients
  • the http-probing patterns come from real federation and client 404s I saw earlier
  • other 404s (federation/v1/hierarchy, key/v2/server, account_data) showed up in my logs but never caused an overflow, so I left them out as you suggested

@liamthexpl0rer: regarding the 403 paths in your rule, please check whether Synapse actually returns those 403s. On my setup, almost all 403s on /_matrix/client/v3/sync were generated by the reverse proxy / bouncer middleware (Traefik OriginStatus: 0), not by Synapse. Whitelisting them would hide real bans.

The new rule is running live on my server now. I'll report back if anything else shows up.

On my homeserver Element got banned by http-probing right after startup.
It resolves reply and thread parents via /rooms/{roomId}/event/{eventId}
(404 for events in rooms I am not joined to) together with
/rooms/{roomId}/relations/{eventId} (403 M_FORBIDDEN from Synapse), and
checks /unstable/org.matrix.msc3814.v1/dehydrated_device (404 without a
dehydrated device).

I added exactly these three requests to the http-probing rule. The 403
is only accepted on the relations endpoint, all other 403s stay out.
New hub test matrix-whitelist-relations covers a whitelisted client and
a client that mixes in a /_synapse/admin request.

Patrick Asmus
support@techniverse.net

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-for-user Waiting for submitter answer regarding specific points in the PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants