Crowdsec views Mattermost clients as attackers

Althought while setting the flag I also updated Mattermost to the latest version.

Yep, that’s perfect. That explains why the feature flag is turned on for you then since you’re using a build based off of the master branch and not a regular packaged release, so that’s good to know. I was worried we might’ve shipped something we didn’t mean to turn on yet. A good thing about that is that you’ll be able to get the fix for this as its released :smiley:

I forgot to mention earlier, but I filed [MM-49863] - Mattermost which seems to be the cause of the issue you’re encountering.

For my own curiosity, did you build that yourself, install it via a package manager, or download a prebuilt server that we made? I’m a bit surprised to see someone running master in the wild, so I’m wondering if there’s a package repository serving up potentially-unstable versions of Mattermost

That´s the official Docker Team Edition. Image on Dockerhub: mattermost/mattermost-team-edition:master

I have gone back to “latest” image, maybe the edge is bleeding a bit much with “master” :smiley:

Yeah, that’d be a good idea. I think latest is the most recent release that’s gone through full release testing and such while master is just the nightly build of the master/main branches of all of our repos which, despite our best efforts, breaks sometimes

1 Like

I am the one who opened this thread a year ago, and have been watching this exchange with quite some interest but thought I had it covered for me.

Well, I just got off a meeting with my users and there were quite a few which complained about Mattermost not working reliably. Works at some point, and then all off a sudden they can’t connect. Works on the phone but not on the PC or the other way round.
In short the typical behaviour one would see with the 4 hour CrowdSecurity block.

I am running Mattermost on Docker and am on v 7.5.2-rc2.

Thought I’d mention that @Twilek is not alone in this and it looks like I need to urgently do something about this.

Hi tomz,

I was waiting for you to chip in - welcome back :slight_smile:
Could you also check your logfiles for the 404s and 403s and share an excerpt with us so we can see if you’re affected by the same requests?

Happy to, here’s both:

sudo cat /var/log/nginx/access.log | grep "404"
46.5. - - [27/Jan/2023:09:07:34 +0100] "GET /static/Metropolis-SemiBold.woff2 HTTP/2.0" 404 19 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Mattermost/4.6.2 Chrome/78.0.3904.130 Electron/7.3.2 Safari/537.36"
46.5. - - [27/Jan/2023:09:07:34 +0100] "GET /api/v4/brand/image?t=0 HTTP/2.0" 404 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Mattermost/4.6.2 Chrome/78.0.3904.130 Electron/7.3.2 Safari/537.36"
46.223. - - [27/Jan/2023:10:00:33 +0100] "GET /static/Metropolis-SemiBold.woff2 HTTP/2.0" 404 19 "https://team.bakdos.de/fdh/channels/aktuelles" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.5249.119 Electron/21.2.0 Safari/537.36 Mattermost/5.2.1"
46.223. - - [27/Jan/2023:10:00:33 +0100] "GET /static/Metropolis-SemiBold.woff2 HTTP/2.0" 404 19 "https://team.bakdos.de/boards" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.5249.119 Electron/21.2.0 Safari/537.36 Mattermost/5.2.1"
46.223. - - [27/Jan/2023:10:00:33 +0100] "GET /static/Metropolis-SemiBoldItalic.woff2 HTTP/2.0" 404 19 "https://team.bakdos.de/fdh/channels/aktuelles" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.5249.119 Electron/21.2.0 Safari/537.36 Mattermost/5.2.1"
87.156"" - - [27/Jan/2023:11:28:10 +0100] "GET /static/products/boards/remote_entry.js HTTP/2.0" 404 19 "-" "Mozilla/5.0 (iPad; CPU OS 16_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Mobile/15E148 Safari/604.1"
87.156"" - - [27/Jan/2023:11:28:10 +0100] "GET /static/3935.aec86521cd5f7c93a364.js HTTP/2.0" 200 40456 "-" "Mozilla/5.0 (iPad; CPU OS 16_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Mobile/15E148 Safari/604.1"
87.156"" - - [27/Jan/2023:11:28:11 +0100] "GET /static/files/code_themes/7654b55b2f3442e914047bab6d9617cb.css HTTP/2.0" 200 1309 "-" "Mozilla/5.0 (iPad; CPU OS 16_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Mobile/15E148 Safari/604.1"
87.156"" - - [27/Jan/2023:11:28:11 +0100] "GET /api/v4/brand/image?t=0 HTTP/2.0" 404 0 "-" "Mozilla/5.0 (iPad; CPU OS 16_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Mobile/15E148 Safari/604.1"
87.156"" - - [27/Jan/2023:11:28:11 +0100] "GET /static/Metropolis-SemiBold.woff2 HTTP/2.0" 404 19 "-" "Mozilla/5.0 (iPad; CPU OS 16_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Mobile/15E148 Safari/604.1"
87.156"" - - [27/Jan/2023:11:28:16 +0100] "GET /static/products/boards/remote_entry.js HTTP/2.0" 404 19 "-" "Mozilla/5.0 (iPad; CPU OS 16_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Mobile/15E148 Safari/604.1"
87.156"" - - [27/Jan/2023:11:28:16 +0100] "GET /static/Metropolis-SemiBold.woff2 HTTP/2.0" 404 19 "-" "Mozilla/5.0 (iPad; CPU OS 16_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Mobile/15E148 Safari/604.1"
87.156"" - - [27/Jan/2023:11:28:16 +0100] "GET /api/v4/brand/image?t=0 HTTP/2.0" 404 0 "-" "Mozilla/5.0 (iPad; CPU OS 16_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Mobile/15E148 Safari/604.1"
87.156 - - [27/Jan/2023:11:28:24 +0100] "GET /static/products/boards/remote_entry.js HTTP/2.0" 404 19 "-" "Mozilla/5.0 (iPad; CPU OS 16_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Mobile/15E148 Safari/604.1"
87.156"" - - [27/Jan/2023:11:28:24 +0100] "GET /api/v4/brand/image?t=0 HTTP/2.0" 404 0 "-" "Mozilla/5.0 (iPad; CPU OS 16_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Mobile/15E148 Safari/604.1"
87.156 - - [27/Jan/2023:11:28:24 +0100] "GET /static/Metropolis-SemiBold.woff2 HTTP/2.0" 404 19 "-" "Mozilla/5.0 (iPad; CPU OS 16_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Mobile/15E148 Safari/604.1"

$ sudo cat /var/log/nginx/access.log | grep "403"
46.223. - - [27/Jan/2023:10:00:33 +0100] "GET /api/v4/trial-license/prev HTTP/2.0" 403 188 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.5249.119 Electron/21.2.0 Safari/537.36 Mattermost/5.2.1"
46.223. - - [27/Jan/2023:10:00:33 +0100] "GET /api/v4/trial-license/prev HTTP/2.0" 403 188 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.5249.119 Electron/21.2.0 Safari/537.36 Mattermost/5.2.1"
46.223. - - [27/Jan/2023:10:00:33 +0100] "GET /api/v4/cloud/products HTTP/2.0" 403 170 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.5249.119 Electron/21.2.0 Safari/537.36 Mattermost/5.2.1"
46.223. - - [27/Jan/2023:10:00:33 +0100] "GET /api/v4/cloud/products HTTP/2.0" 403 170 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.5249.119 Electron/21.2.0 Safari/537.36 Mattermost/5.2.1"

The 403 seem to all come from my machine.
Focalboard is enabled.

Let me know if I can provide any other detail.

How did you do that?

Thanks.
For the fonts missing (Metropolis-*.woff2), there’s already a ticket:
https://mattermost.atlassian.net/browse/MM-41813

The calls to /api/v4/brand/image are OK I guess, because that’s how the application tries to identify if a custom brand logo is available. The returncode 404 is expected when there is none:
https://api.mattermost.com/#tag/brand/operation/GetBrandImage

Not sure about /static/products/boards/remote_entry.js, this is also related to the boards plugin - let me see if I can find out more about it.

I got a few more errors in yesterdays log but can’t upload a txt file and it seems to be too much text to add it to the post.

Actually I’m just interested in the URLs and how often they’ve been requested.
Assuming that the format of your logfile is identical to the one you already posted, could you please run this command against your logfile and post the results?

$ sed -n 's!^.*"GET \(.\+\) HTTP/[^"]\+" 404.*!\1!p' /path/to/your/logfile.txt | sort | uniq -c | sort -rn
      6 /static/Metropolis-SemiBold.woff2
      4 /api/v4/brand/image?t=0
      3 /static/products/boards/remote_entry.js
      1 /static/Metropolis-SemiBoldItalic.woff2

This is the resulting output for your example logfile and lists the number of occurences per URL.

Those are not happening anymore on 7.7.1, just verified that - so if the server gets upgraded to the latest version, they will go away.

Here you go:

 35 /static/Metropolis-SemiBold.woff2
 12 /static/Metropolis-SemiBoldItalic.woff2
  9 /static/products/boards/remote_entry.js
  7 /api/v4/brand/image?t=0

Were those the ones that led to the blockage?

I can see that 7.7.1 is available to me, I can upgrade later this evening. Anything specific I should look out for after the upgrade is done? E.g. no more remote_entry.js ?

None of these should cause a block actually, but I’m not sure how crowdsec works and why it blocks you (if this is actually the case).
The assumption made in this thread here was that this is happening because of the 404s and 403s and that’s what we are looking at right now.
Upgrading to 7.7.1 will only fix the 404’s on remote_entry.js - all other issues will still be there then and need to be ignored for the time being.

Before upgrading, please read the important upgrade notes which might contain relevant information for the version you’re currently running, but besides that, everything should run smooth.

Sounds good, thanks!
On to the upgrade…and done! I’ll have an eye on my log in the next couple of days.
Also told my users to immediately alert me i they experience logon problems, we’ll see.

Have a nice weekend…

Awesome - if it’s really crowdsec, you’d have to investigate their logs, I’m unfamiliar with this application so I’m not sure if I can help with that, sorry.

The way they work is that they parse logs, in my case /var/log/nginx/error.log, look for common attacks to block locally, or if an IP address has been reported by anyone in the network (like my server) they block the IP right away.
So yes, I need to keep an eye on their logs and see who got blocked and why.

I am using Traefik with crowdsec as a middleware. I changed it from being a general middleware for all http and https traffic to a middleware on a per container basis and then removed it for mattermost.