Issue after server upgrade & MM 10.3 upgrade

Troubleshooting
1

Summary
After an upgrade Ubuntu to 24.04.1 on Digital Ocean, and an upgrade of Mattermost to 10.3, some users cannot get Mattermost to load on desktop app, and web loading fails.

Expected behavior
I operate two instances of Mattermost, both at Digital Ocean, both originally running on Ubuntu 22.04. I recently updated both servers to Ubuntu 24.04.1 and updated both Mattermost instances to 10.3. One of them is running perfectly, as I’d expect! The other one is not.

Observed behavior
When visiting the URL of the malfunctioning instance the usual “loading” screen is replaced by a giant graphic, and nothing further happens.

Screenshot 2025-01-06 at 2.57.53 PM
Screenshot 2025-01-06 at 2.57.53 PM1920×1138 24.8 KB

The JS console shows “Failed to load resource: zero byte resource” for /static/css/initial_loading_screen.css and /static/manifest.json, as well as a “network connection was lost” error.

The weirdest part is that my desktop app is working perfectly, including through restarts and reloads, but I have one colleague who gets that same graphic in her desktop app.

The logs are not giving me much info:

Jan 06 20:35:34 chat mattermost[4352]: {"timestamp":"2025-01-06 20:35:34.677 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:185","method":"POST","url":"/api/v4/client_perf","request_id":"zgz9gci1qtb18rb3e13pqdiacc","user_id":"yypxpbuq8f87fyeoz7tdiywb8w"}
Jan 06 20:35:34 chat mattermost[4352]: {"timestamp":"2025-01-06 20:35:34.678 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:185","method":"POST","url":"/api/v4/users/status/ids","request_id":"95tawein6pni8rib3pt4nkmqhc","user_id":"yypxpbuq8f87fyeoz7tdiywb8w","status_code":"200"}
Jan 06 20:35:34 chat mattermost[4352]: {"timestamp":"2025-01-06 20:35:34.708 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:185","method":"GET","url":"/api/v4/plugins/webapp","request_id":"6md9fwcm6fbotgqe8jmgiezdhc","user_id":"yypxpbuq8f87fyeoz7tdiywb8w","status_code":"200"}
Jan 06 20:35:34 chat mattermost[4352]: {"timestamp":"2025-01-06 20:35:34.714 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:185","method":"GET","url":"/api/v4/users/me/teams/ka475a9qdbde5pgq3akjmmyf3r/channels/members","request_id":"4n65rd885j8bxrxo95nwkty6oe","user_id":"yypxpbuq8f87fyeoz7tdiywb8w","status_code":"200"}
Jan 06 20:35:34 chat mattermost[4352]: {"timestamp":"2025-01-06 20:35:34.715 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:185","method":"GET","url":"/api/v4/users/me/teams/ka475a9qdbde5pgq3akjmmyf3r/channels","request_id":"3giofj7rk7brjyxgtmxw4d9wda","user_id":"yypxpbuq8f87fyeoz7tdiywb8w","status_code":"200"}
Jan 06 20:35:34 chat mattermost[4352]: {"timestamp":"2025-01-06 20:35:34.717 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:185","method":"GET","url":"/api/v4/users/me/teams/unread","request_id":"br6z81wpqb8x5ddbmjaggc6gnc","user_id":"yypxpbuq8f87fyeoz7tdiywb8w","status_code":"200"}
Jan 06 20:35:34 chat mattermost[4352]: {"timestamp":"2025-01-06 20:35:34.720 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:185","method":"GET","url":"/api/v4/channels/39oqzq98mjy69fteisgfchqoow/posts","request_id":"scs7ps649p8cbph6ko7apubfwy","user_id":"yypxpbuq8f87fyeoz7tdiywb8w","status_code":"200"}
Jan 06 20:35:34 chat mattermost[4352]: {"timestamp":"2025-01-06 20:35:34.734 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:185","method":"POST","url":"/api/v4/users/ids","request_id":"394g9r93tbd1xy6qkkfqs4x69e","user_id":"yypxpbuq8f87fyeoz7tdiywb8w","status_code":"200"}
Jan 06 20:35:38 chat mattermost[4352]: {"timestamp":"2025-01-06 20:35:38.684 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:185","method":"POST","url":"/api/v4/client_perf","request_id":"oi4fnrbtgidwmeru5p6rek9k3c","user_id":"mhbczwyt9jgyxpedmrhbxrzorc"}
Jan 06 20:35:38 chat mattermost[4352]: {"timestamp":"2025-01-06 20:35:38.686 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:185","method":"POST","url":"/api/v4/users/status/ids","request_id":"ddx1u7jgs7np8fem9hrdg75wnr","user_id":"mhbczwyt9jgyxpedmrhbxrzorc","status_code":"200"}

I’ve stopped and restarted Mattermost and I’ve rebooted the server. I’m honestly not sure what else to try. Any advice would be hugely appreciated!

2

I wound up restoring from backup due to increasingly flaky server behavior, and all is now well.

1 Like
3

What version did you upgrade from?

4

10.0 And I have to take my “all is well” back – it was working fine last night but this morning it’s back to the same behavior. If you have any ideas of what I should look at, I’d be most grateful.

5

Are you going through any kind of proxy or load balancer?

6

No, no proxy or load balancer. Just nginx on Ubuntu (now back to 22.04 since restoring from back up).

7

Do the nginx logs tell you anything?

8

A few things after an nginx restart:

Jan 07 14:07:13 chat nginx[18523]: nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2">
Jan 07 14:07:13 chat nginx[18523]: nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2">
Jan 07 14:07:13 chat systemd[1]: nginx.service: Can't open PID file /run/nginx.pid (yet?) after start: Operation>
9

I’d fix those warnings on general principle.

Maybe increase verbosity for nginx, tail -f that log, and see if anything comes up?

Maybe take nginx out of the mix and hit Mattermost directly as a test?

10

Hmmmmm. Looks like it could be a permissions issue?

2025/01/07 14:56:42 [crit] 18526#18526: *441 open() "/var/cache/nginx/b/6b/ca86294c468fe9a9e2cc4b17d35fd6bb.0000000035" failed (13: Permission denied) while reading upstream, client: 23.121.165.145, server: chat.meshresearch.net, request: "GET /static/manifest.json HTTP/2.0", upstream: "http://127.0.0.1:8065/static/manifest.json", host: "chat.meshresearch.net"

/opt/mattermost/client/manifest.json is currently set to -rw-r--r-- – what should it be?

11

Huh. Seems like that should make it never work at all! Do you have SELinux enabled, or AppArmor, or anything else? And are you sure that http://127.0.0.1:8065/static/manifest.json is touching /opt/mattermost/client/manifest.json?

12

I don’t have either SELinux or AppArmor enabled. And I’m not at all sure that http://127.0.0.1:8065/static/manifest.json is touching /opt/mattermost/client/manifest.json! In fact, config.json doesn’t seem to be configured – the listen address is set to 8065 but not many of the other service settings are complete. I’m wondering if the upgrade (done via apt-get) might have overwritten that file?

13

How are you deploying? Installed directly on Ubuntu, or in a Docker container, or ???

14

Directly on Ubuntu via the “Deploy on Ubuntu” (e.g., apt-get) instructions.

15

Can you access Mattermost directly and not via nginx, just to see if that helps us to localize the issue?

How did you perform the upgrade? Did you use Upgrade Mattermost Server - Mattermost documentation ? Part of those instructions moves the old /opt/mattermost to a backup directory, so you should be able to recover config.json if it was overwritten.

16

Unfortunately, no – I used the update instructions here: https://docs.mattermost.com/install/install-ubuntu.html. And I’m afraid I’m not sure how to access Mattermost directly. Stopping and restarting nginx several times seems to have shaken something loose, and I can now access the web app, but I’m worried that whatever’s gone awry is going to keep causing failures.

EDITED TO ADD: The good news, I guess, is that the problem seems to be nginx? So I’m going to see if I can troubleshoot that.

17

Hmm. I’d imagine using apt to upgrade shouldn’t overwrite files. And if config.json was overwritten, you’d lose your data source. I suspect this is something in nginx. I’d recommend tracking down and fixing the issues indicated in the logs.

Does nginx run on the same instance as Mattermost? I visited https://chat.meshresearch.net and see the issue you’re talking about. You might try allowing your client to access 8065 through the firewall and see if http://chat.meshresearch.net:8065 works. I bet it does.

1 Like
18

Okay! I’ve spent the last little bit reconfiguring nginx using the info on Set up an NGINX proxy - Mattermost documentation, and it seems to be working. (Mostly! I’m getting a Javascript error in Firefox, but I’ll deal with that later.) I’ll cross my fingers that it stays stable!

1 Like
19

If it’s an nginx issue, I am wondering how did upgrading Mattermost change your nginx settings.