Bitnami-Mattermost 7.x - 502 Bad Gateway

For troubleshooting questions, please post in the following format:

Summary
502 Bad Gateway

Steps to reproduce
Navigate to the Mattermost login URL

Expected behavior
Display login screen

Observed behavior
All of a sudden our on-premise hosted Mattermost server (bitnmai-mattermost
-7.1.1) stopped working. We have checked the conf files, the fw, the SSL certs, all seems to be in order.

The nginx error log shows upstream errors (screenshot attached).

Mattermost.conf:

upstream backend {
    server 127.0.0.1:8065;
}

server {
   listen         80 default_server;
    server_name    172.16.11.226;
   return         302 https://chat.xzfv.ad:443$request_uri;
}

proxy_cache_path /opt/bitnami/nginx/cache levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;

server {
   listen 443 ssl;
    server_name    172.16.11.226;

   ssl on;
#   ssl_certificate /opt/bitnami/apps/mattermost/conf/certs/server.crt;
#   ssl_certificate_key /opt/bitnami/apps/mattermost/conf/certs/server.key;
   ssl_certificate /etc/ssl/private/chat.xzfv.ad.cer;
   ssl_certificate_key /etc/ssl/private/chat.xzfv.ad.key;
   ssl_session_timeout 5m;
   ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
   ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
   ssl_prefer_server_ciphers on;
   ssl_session_cache shared:SSL:10m;

   location /api/v4/websocket {
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
      proxy_set_header X-Forwarded-Ssl on;
      client_max_body_size 50M;
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Frame-Options SAMEORIGIN;
      proxy_buffers 256 16k;
      proxy_buffer_size 16k;
      proxy_read_timeout 600s;
      proxy_pass http://backend;
   }

   include "/opt/bitnami/apps/bitnami/banner/conf/banner-substitutions.conf";

location ~ /plugins/focalboard/ws/* {
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       client_max_body_size 50M;
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       client_body_timeout 60;
       send_timeout 300;
       lingering_timeout 5;
       proxy_connect_timeout 90;
       proxy_send_timeout 300;
       proxy_read_timeout 90s;
       proxy_pass http://backend;
   }




   location / {
      proxy_set_header X-Forwarded-Ssl on;
      client_max_body_size 50M;
      proxy_set_header Connection "";
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Frame-Options SAMEORIGIN;
      proxy_buffers 256 16k;
      proxy_buffer_size 16k;
      proxy_read_timeout 600s;
      proxy_cache mattermost_cache;
      proxy_cache_revalidate on;
      proxy_cache_min_uses 2;
      proxy_cache_use_stale timeout;
      proxy_cache_lock on;
      proxy_pass http://backend;
    }
    include "/opt/bitnami/apps/bitnami/banner/conf/banner.conf";
}



Not sure where to go with it at this point.

Any help appreciated!

Hi st84,

welcome to the mattermost forums!

I’ve downloaded the bitnami 7.1.1 OVA and deployed it in my environment to see how this is working and from what I can see in your logs it seems as if the mattermost application server is not working anymore.

When you login to the console of this system, can you please check the logfile /opt/bitnami/mattermost/logs/mattermost.log for hints on what’s going on? When everything is working, one of the last lines should be “Server is listening on …” right after the start of the service.

You should also see the mattermost process running, which probably isn’t the case on your system right now, you can check that by running:

ps ax | grep mattermost

If no mattermost process is running and you cannot find a reason for the problems in the log, you can also try to manually start the server so you get the output directly on the terminal. To do that, while connected via SSH or console to your bitnami VM, run the following series of commands:

sudo -u mattermost -s
cd /opt/bitnami/mattermost
bin/mattermost

Hi agriesser,

I found the issue. For some, to me unknown reason, the config.json was missing two closing ‘{’ at the end.

It’s very puzzling that it worked up until a few days ago, and then just stopped.
Anyways, thank for your help! This helped finding the issue:

sudo -u mattermost -s
cd /opt/bitnami/mattermost
bin/mattermost

Awesome, thanks for sharing the solution with us and great to see that you’re up and running again :slight_smile: