Nginx 502 bad gateway errors after Mattermost docker update

Troubleshooting
1

Hi. I’ve been running Mattermost in a jelastic-based docker instance for about a year now – I’m a complete docker neophyte, but things have gone well and typically upgrades have been easy.

A community that I’m not the host of updated to 5.33 today, and so I went into my instance to do the same. All appeared fine, but now I’m throwing nginx 502 bad gateway errors that I can’t resolve. When I check the Mattermost version at the command line, I get a huge pile of errors, including the info that my software version (5.30) and my db schema version (5.32) don’t match. I’m not sure if that’s causing the current problem, but it seems like it needs resolution in any case. Any ideas would be welcome!

2

Next question: buried in among the pile of messages were a couple of errors indicating that SiteURL and SMTP info were not set. And in exploring /mattermost-docker/volumes/app/mattermost/config/config.json, it indeed appears that the file has been overwritten with a default – no SiteURL, no SMTP info, etc. Is there any chance of reverting to a pre-update version of that file…?

3

More info: I’ve managed to get in and begin reconstructing the config settings, including SiteURL, SMTP, etc. Those settings require a server restart to take effect. However, every time I restart the server those settings disappear again. I’m… a little beside myself.

4

Can you confirm that you have write permissions to the configuration file that you are making changes to? @ahmaddanial I think you would be better off providing suggestions related to the docker side of things, I’m not as experienced with Docker, however I can help with the 502 code part of things.

5

Thanks – I do have write permissions, and I’ve confirmed that the changes have been written before the restart. I’m in touch with the hosting provider at this point and am asking for things to be restored to a pre-update state. Fingers crossed!

6

Also FYI, v5.33.1 was released today to fix some websocket issues.

7

Basically th2 502 HTTP error means its a bad gateway, which could be at any point in your connection to the server, not specifically related to the configuration of the Mattermost installation itself. Its possible that it is caused from an upstream issue such as an authoritative domain name server going offline for your domain, a failure of one of your current DNS servers to resolve everything to your domain/server properly, etc. I would suggest checking your error.log file for Nginx in /var/log/ or /var/log/nginx/ to see if there are more details that can be determined from there.

8

Hi @kfitz

As @amy.blais mentioned, the release today helps to fix the websocket issue based on the changelog here:

  • Fixed an issue where WebSockets failed with TLS connections. MM-34000
  • Fixed a race condition which would crash the app server due to improper handling of WebSockets closing. MM-33233

As for this one:

Sounds like a known issue related to the existing bugs we have here:

  • Config.json can reset when running the command systemctl restart mattermost , and when running any commands that write to the config (e.g. config or plugin ) MM-33752, MM-32390.
9

Thanks for this, @ahmaddanial – it definitely explains the config.json resets I was seeing!

@XxLilBoPeepsxX I’ve looked into the logs and am not finding anything at all helpful, alas.

This morning I cloned the environment and in the clone updated to 5.33.1 – I think: running docker exec -it mattermost-docker_app_1 mattermost version produces a HIGHLY verbose set of messages (all the way down to “info”), but buried in there is

Version: 5.30.0
Build Number: 5.31.0
Build Date: Fri Jan 15 02:36:16 UTC 2021
Build Hash: 5c6f37ee6db78c8440145a27320baebdcc6e8311
Build Enterprise Ready: true
DB Version: 5.32.0

The 502s didn’t go away, so in the cloned environment I took down the load balancer entirely… and am still getting nginx 502s. So… I’m at a loss.

My next thought is to try starting over with a clean environment and a fresh installation, and then importing the data from the old environment, but running the CLI bulk exporter doesn’t seem to be working. I get the same crazy verbose overflow of messages, but no output file despite there not being errors indicating that the process has failed. I’m just flummoxed.

Thanks for your help so far, all – I really appreciate it!

1 Like
10

Here’s what I think is the key bit from the bulk exporter process:

{“level”:“info”,“ts”:1616517800.657653,“caller”:“app/export.go:57”,“msg”:“Bulk export: exporting version”}
{“level”:“info”,“ts”:1616517800.658037,“caller”:“app/export.go:62”,“msg”:“Bulk export: exporting teams”}
{“level”:“info”,“ts”:1616517800.660892,“caller”:“app/export.go:67”,“msg”:“Bulk export: exporting channels”}
{“level”:“info”,“ts”:1616517800.6653361,“caller”:“app/export.go:72”,“msg”:“Bulk export: exporting users”}
{“level”:“info”,“ts”:1616517800.7146847,“caller”:“app/export.go:77”,“msg”:“Bulk export: exporting posts”}
{“level”:“info”,“ts”:1616517915.9105906,“caller”:“app/export.go:82”,“msg”:“Bulk export: exporting emoji”}
{“level”:“info”,“ts”:1616517915.9117804,“caller”:“app/export.go:87”,“msg”:“Bulk export: exporting direct channels”}
{“level”:“info”,“ts”:1616517915.9171648,“caller”:“app/export.go:92”,“msg”:“Bulk export: exporting direct posts”}
{“level”:“info”,“ts”:1616517918.6002073,“caller”:“app/server.go:721”,“msg”:“Stopping Server…”}
{“level”:“info”,“ts”:1616517918.6006017,“caller”:“app/web_hub.go:115”,“msg”:“stopping websocket hub connections”}
{“level”:“info”,“ts”:1616517918.60092,“caller”:“app/plugin.go:300”,“msg”:“Shutting down plugins”}
{“level”:“warn”,“ts”:1616517918.6058302,“caller”:“plugin/hclog_adapter.go:69”,“msg”:“error closing client during Kill”,“plugin_id”:“com.mattermost.plugin-channel-export”,“wrapped_ext
ras”:“errunexpected EOF”}
{“level”:“warn”,“ts”:1616517918.6060085,“caller”:“plugin/hclog_adapter.go:71”,“msg”:“plugin failed to exit gracefully”,“plugin_id”:“com.mattermost.plugin-channel-export”}
{“level”:“debug”,“ts”:1616517918.605957,“caller”:“plugin/hclog_adapter.go:51”,“msg”:“plugin process exited”,“plugin_id”:“com.mattermost.plugin-channel-export”,“wrapped_extras”:“pathp
lugins/com.mattermost.plugin-channel-export/server/dist/plugin-linux-amd64pid22938”}

1 Like
11

@ahmaddanial I think that this entry right here is the one we are looking for. The error
wrapped_extras”:“errunexpected EOF seems to me to be the contributing factor.

By any chance could you suggest a further step to determine where/what is causing the error, or what the next step we should proceed with is?