Repository navigation
Conversation
|
Hello, Thanks for the PR. Unfortunately, I think using a parser whitelist for that is a bit too broad: you are effectively ignoring every request on anything that starts with those paths. A much better solution would be to use a postoverflow to only allow those paths for specific scenarios. |
|
Thank you very much for your feedback. |
|
Hi, I'm facing the same issue, then came here, I did follow the link about postoverflow, here is a file I'm now trying name: synapse-client-whitelists
description: Whitelist synapse client application for crawl non static
whitelist:
reason: synapse-client can trigger FP
expression:
- evt.Overflow.Alert.Scenario == "crowdsecurity/http-crawl-non_statics" and all(evt.Overflow.Alert.Events, {.GetMeta("http_path") startsWith "/_matrix/client/"})
- evt.Overflow.Alert.Scenario == "crowdsecurity/http-probing" and all(evt.Overflow.Alert.Events, {.GetMeta("http_path") startsWith "/_matrix/client/"})I did add the http-probing scenario because in some rooms there are a lot of 404 😕 that trigger the ban [edit] |
|
The suggested postoverflow didn't work for me. Part of it is that some 404s caused by Nextcloud activity seem to be causing the bucket to overflow and then everything goes sideways. Based on what I'm seeing, a parser whitelist may be the better option, if there are other services that may also be generating 404s... Nextcloud is notorious for being bad about 404s... even with https://app.crowdsec.net/hub/author/crowdsecurity/configurations/nextcloud-whitelist . |
this PR is the whitelist about matrix/synapse |
There was a problem hiding this comment.
Pull request overview
Adds a new enrich-stage HTTP whitelist parser intended to prevent legitimate Matrix/Synapse endpoints from being counted toward generic HTTP scanning/crawling scenarios (e.g., http-probing, http-crawl-non_statics).
Changes:
- Introduces a Matrix/Synapse path-prefix whitelist parser (
/_matrix/,/_synapse/,/.well-known/matrix/). - Adds documentation describing why these endpoints commonly generate benign “suspicious-looking” HTTP patterns.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| parsers/s02-enrich/crowdsecurity/matrix-whitelist.yaml | Adds a whitelist parser matching Matrix/Synapse request paths. |
| parsers/s02-enrich/crowdsecurity/matrix-whitelist.md | Documents the rationale and covered Matrix/Synapse endpoints. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -0,0 +1,9 @@ | |||
| name: my/matrix-whitelist | |||
There was a problem hiding this comment.
name is set to my/matrix-whitelist, but all other parsers in parsers/s02-enrich/crowdsecurity/ use the crowdsecurity/ namespace (e.g. parsers/s02-enrich/crowdsecurity/actual-budget-whitelist.yaml:1). Using a different namespace is likely to fail hub conventions/linting and makes the parser harder to discover; rename it to crowdsecurity/matrix-whitelist (and keep it consistent with the file path).
| name: my/matrix-whitelist | |
| name: crowdsecurity/matrix-whitelist |
| @@ -0,0 +1,9 @@ | |||
| name: my/matrix-whitelist | |||
| description: "Whitelist Matrix/Synapse requests from NPMplus logs" | |||
There was a problem hiding this comment.
The description says "from NPMplus logs", but the filter matches any event with evt.Meta.service == 'http' and the standard http_access-log/http_error-log log types. Either scope the filter to NPMplus-specific events (if that’s the intent) or update the description to reflect that this whitelist applies to generic HTTP logs.
| description: "Whitelist Matrix/Synapse requests from NPMplus logs" | |
| description: "Whitelist Matrix/Synapse HTTP requests from generic http_access-log/http_error-log entries" |
| - "evt.Meta.http_path startsWith '/_matrix/'" | ||
| - "evt.Meta.http_path startsWith '/_synapse/'" | ||
| - "evt.Meta.http_path startsWith '/.well-known/matrix/'" |
There was a problem hiding this comment.
This whitelists all requests to /_matrix/ and /_synapse/ purely by path prefix. Unlike other HTTP app whitelists in this directory (which typically narrow by verb/status and specific endpoints), this creates a broad blind spot where probing/bruteforce against Synapse/Matrix endpoints would no longer contribute to remediation decisions. Consider narrowing the whitelist conditions (e.g., to specific verbs/statuses/endpoints known to FP) or moving the exception into the specific scenarios that FP (so Matrix-specific attacks can still be detected).
| - "evt.Meta.http_path startsWith '/_matrix/'" | |
| - "evt.Meta.http_path startsWith '/_synapse/'" | |
| - "evt.Meta.http_path startsWith '/.well-known/matrix/'" | |
| - "evt.Meta.http_verb == 'GET' && evt.Meta.http_path startsWith '/.well-known/matrix/'" |
| name: my/matrix-whitelist | ||
| description: "Whitelist Matrix/Synapse requests from NPMplus logs" | ||
| filter: "evt.Meta.service == 'http' && evt.Meta.log_type in ['http_access-log', 'http_error-log']" | ||
| whitelist: | ||
| reason: "Matrix federation/client traffic" | ||
| expression: | ||
| - "evt.Meta.http_path startsWith '/_matrix/'" | ||
| - "evt.Meta.http_path startsWith '/_synapse/'" | ||
| - "evt.Meta.http_path startsWith '/.well-known/matrix/'" |
There was a problem hiding this comment.
A new whitelist parser is added, but there is no corresponding .tests/matrix-whitelist/ (most existing whitelists here have dedicated tests, e.g. .tests/actual-budget-whitelist/, .tests/calibre-web-whitelist/, .tests/nextcloud-whitelist/). Add a minimal test fixture asserting that Matrix/Synapse paths are marked whitelisted and that non-Matrix paths are not.
Sorry, I wasn't clear enough. The combination of matrix/synapse and Nextcloud endpoints 404'ing made the postoverflow whitelist to fail. Or really, if at any point any other 404 happens during that window, that would make the postoverflow whitelist to fail thanks to The contents of this PR do seem to work for me, even if allows for a lot of request URLs. |
|
I'm also using Synapse and getting banned due to the scenario crowdsecurity/http-probing. |
|
@pscriptos Is there any progress? I have the same problem and it would be very nice if this can be merged soon. |
Yesterday I applied this rule to my system and it works too. If @pscriptos does not react, can you maybe open an new PR @Nemric? |
|
Hi, This PR still needs some attention. And some whitelists seem a bit broader than mandatory. I think this should be done by someone using matrix/synapse. regards, |
I'm open to investigate if someone can tell me what I should test. You can see my current NixOS configuration at Codeberg (links below). ;) @sabban Do you have an idea for a rule that I can test? |
|
I also found the official API specifications for the matrix protocol. |
|
Hi everyone, sorry for the long silence on this one, and thanks for all the feedback and testing. I've picked this up again and agree with @blotus and @sabban: the parser whitelist is too broad. It drops every event under Looking at my logs, the actual false positives come from a very small set of legitimate 404s:
Because every room or event ID is a distinct path, these trip What I've changed locally and am currently testing:
First results look good: within the first minutes it already prevented several bans of federating homeservers. I'll let it run for a few days, then update this PR accordingly (correct namespace, description, hub tests under Thanks for your patience! |
|
Hi @pscriptos, thanks for picking this up again, and thanks @Nemric, @liamthexpl0rer and @gemarcano for testing. Your plan (a postoverflow scoped to the scenarios that false-positive) is the right direction, as @blotus suggested. To keep it as narrow as possible:
Something like this works in a quick hubtest on my side: name: whatever-fits-you/matrix-whitelist
description: "Whitelist known false positives from Matrix/Synapse clients and federation"
whitelist:
reason: "Matrix/Synapse known false positives"
expression:
- |
evt.Overflow.Alert.GetScenario() == 'crowdsecurity/http-probing' && all(evt.Overflow.Alert.Events, {
.GetMeta('http_verb') == 'GET' && .GetMeta('http_status') == '404' && (
.GetMeta('http_path') matches '^/_matrix/client/(r0|v3)/rooms/[^/]+/state/m\\.room\\.encryption/?$' ||
.GetMeta('http_path') matches '^/_matrix/federation/v1/event/[^/]+$'
)
})If For tests, add About @gemarcano's point: with @liamthexpl0rer, if you can run this against your own logs and report which false positives remain, that would help a lot. Regards, |
|
Hello, I’m also trying to build filters for matrix/synapse, here a some cases that I don’t think are covered by the more "restricted" rules: Synapse’s media serving is not detected as "static" by the native filter: "evt.Meta.service == 'http' && evt.Meta.log_type in ['http_access-log', 'http_error-log'] && evt.Meta.http_path startsWith '/_matrix/client/v1/media/'"
statics:
- target: evt.Parsed.static_ressource
value: 'true'With regards to the
|
This comment was marked as outdated.
This comment was marked as outdated.
|
I've found some other false-positives and extended the whitelist. I will wait one week if there are still any alerts and post the whitelist then. ^^ |
I replaced the broad parser whitelist with a postoverflow that only discards http-probing and http-crawl-non_statics overflows when all events are known harmless Matrix/Synapse requests (method, status and exact path), and added hub tests for both scenarios.
|
Thanks a lot @sabban for the detailed guidance, and @plule and @liamthexpl0rer for the additional data. I've updated the PR accordingly:
Evidence from my own setup (Synapse behind Traefik, ~4 days of logs):
@liamthexpl0rer: regarding the The new rule is running live on my server now. I'll report back if anything else shows up. |
On my homeserver Element got banned by http-probing right after startup.
It resolves reply and thread parents via /rooms/{roomId}/event/{eventId}
(404 for events in rooms I am not joined to) together with
/rooms/{roomId}/relations/{eventId} (403 M_FORBIDDEN from Synapse), and
checks /unstable/org.matrix.msc3814.v1/dehydrated_device (404 without a
dehydrated device).
I added exactly these three requests to the http-probing rule. The 403
is only accepted on the relations endpoint, all other 403s stay out.
New hub test matrix-whitelist-relations covers a whitelisted client and
a client that mixes in a /_synapse/admin request.
Patrick Asmus
support@techniverse.net
Description
Postoverflow whitelist for known false positives caused by legitimate Matrix/Synapse traffic. It replaces the original parser whitelist from this PR, which was too broad.
An overflow is only discarded when it comes from one of the scenarios below and all of its events are known harmless requests (method, status and exact path). Any other request in the same overflow (e.g.
GET /_matrix/.env) still raises the alert, and all other scenarios keep seeing Matrix traffic.crowdsecurity/http-probingGET→404on/_matrix/client/(r0|v3)/rooms/{roomId}/state/m.room.encryptioncrowdsecurity/http-probingGET→404on/_matrix/federation/v1/event/{eventId}crowdsecurity/http-crawl-non_staticsGET→200/304on/_matrix/client/v1/media/thumbnail/{server}/{mediaId}Not whitelisted on purpose:
/login,/_synapse/(admin API), non-GETrequests and403responses.Note: with Traefik,
http_pathincludes the query string, so the path regexes allow an optional?...suffix.Checklist
.tests/matrix-whitelist,.tests/matrix-whitelist-crawl)cscli hubtest runlocally with CrowdSec v1.8.2)