For the last couple of weeks I have been getting user complaints about not being able to logon. Highly erratic behaviour where login wasn’t possible with service not found messages and a few hours later things would work again.
Well, to cut a somewhat long troubleshooting story short, it turned out to be my Intrusion Protection System, https://crowdsec.net/ which all of a sudden views Mattermost clients looking for backend services as attackers and blocks the IP for 4 hours.
sudo cscli decisions list
+----------+----------+-------------------+--------------------------------------+--------+---------+----------
| ID | SOURCE | SCOPE:VALUE | REASON | ACTION
| 11870524 | crowdsec | Ip:149.172.x.y | crowdsecurity/http-crawl-non_statics | ban
Now, I don’t know if this behaviour is due to some recent changes in how MM clients communicate or if it’s Crowdsec which changed. I will take this up with Crowdsec but I thought I make you aware.