502 Gateway after renewing cert

Summary

Renewing expired LetsEncrypt cert results in 502 Bad Gateway

Steps to reproduce

docker-compose stop
sudo certbot certonly
docker-compse up
Refresh Mattermost web client

Expected behavior

Should see the usual web interface

Observed behavior

502 Gateway error

The docker logs contained

web_1  | 142.90.32.247 - - [02/Apr/2021:14:02:16 +0000] "GET / HTTP/2.0" 502 552 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" "-"
web_1  | 2021/04/02 14:02:16 [error] 11#11: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 142.90.32.247, server: , request: "GET / HTTP/2.0", upstream: "http://172.20.0.3:8000/", host: "mattermost.techsmarts.dev"

I don’t see where http://172.20.0.3:8000 comes from? The private address is 172.31.16.223.

Also, docker ps shows “unhealthy” for the app container. But logs only show

Using existing config file /mattermost/config/config.json
Using existing database connection
Starting mattermost

Does anyone know what the last known working version of Mattermost is? I see that GitHub - mattermost/mattermost-docker: Dockerfile for mattermost in production is currently broken. I have gone back as far as 5.30 but the instructions in the README do not work. I am trying to switch from https to HTTP to bypass any possible TLS issues but get stuck in an endless redirect loop back to https. Now thinking of downgrading even further to maybe 5.29 to see if that version works.

Can I ask what this IP address is? Is this the IP address of your web server, or of yourself?

My research shows that the origin IP address of your web server is 35.183.206.164 and has the ports open as follows:

PORT     STATE  SERVICE
22/tcp   open   ssh
80/tcp   open   http
443/tcp  open   https
8000/tcp closed http-alt
8001/tcp closed vcom-tunnel

What I have noticed, however, is that you have an automatic HTTP to HTTPS redirect on the domain as well. I use certbot myself as well, both for Mattermost as well as production websites, and I have found that you must disable these automatic HTTPS redirects because certbot authenticates domain ownership by placing a file in the webroot of the file system, and then making an external HTTP request to then view that file. If you have your domain automatically redirecting to HTTPS requests only, it will not be able to authenticate ownership. You don’t need to keep the HTTPS redirect disabled permanently, it is just for the reauthentication of the domain ownership, then you should be fine to re-enable it.

1 Like

The problem has been solved, albeit with a brute-force solution. I backed up the contents of the /volumes directory then created a new EC2 instance. I installed Mattermost following these steps and not the confusing and ultimately incorrect steps in the repo README. Then restored the /volumes directory

git clone https://github.com/mattermost/mattermost-docker.git
cd mattermost-docker
git checkout release-5.30 # There is a significant known Postgres issue with 5.31
docker-compose build
mkdir -pv ./volumes/app/mattermost/{data,logs,config,plugins,client-plugins}
sudo chown -R 1000:1000 ./volumes/app/mattermost/ # Should match app:args:PUID in docker-compose.yml
docker-compose up -d
certbot certonly --standalone -d {domain-name}
cp {path-to-certbot}/cert.pem volumes/web/cert 
cp {path-to-certbot}/privkey.pem volumes/web/cert/key-no-password.pem
1 Like

Well, while I’m glad that the issue has been resolved for the time being, in the end, from what information I was able to find when the new certificate is up for renewal unless you change some of your HTTP server implementation, you will without a doubt, 100% for sure encounter the same issue again. Nonetheless, this method works, and as such, you can continue to use it down the road, of course.