Mattermost service not starting after upgrade

Troubleshooting
1

Summary
After upgrading Mattermost from v8.1.0 to v9.7.1 on Ubuntu server 22.04.4 LTS, I’m unable to start the Mattermost service again.

Steps to reproduce
systemctl stop mattermost.service
sudo apt update -y
sudo apt upgrade

I’m using PostgreSQL. The database service is up and running normally.

Expected behavior
with systemctl restart mattermost.service the server should be up and working again. I haven’t made any changes to the database or user/group rights on files/folders.

Observed behavior
systemctl status mattermost.service outputs the following:

○ mattermost.service - Mattermost
     Loaded: loaded (/lib/systemd/system/mattermost.service; enabled; vendor preset: enabled)
     Active: inactive (dead) (Result: exit-code) since Wed 2024-04-24 19:32:21 UTC; 11h ago
    Process: 22814 ExecStart=/opt/mattermost/bin/mattermost (code=exited, status=1/FAILURE)
   Main PID: 22814 (code=exited, status=1/FAILURE)
        CPU: 6.276s

Apr 24 19:32:21 ov-9a01db systemd[1]: Stopped Mattermost.
Apr 24 19:32:21 ov-9a01db systemd[1]: mattermost.service: Consumed 6.276s CPU time.

when I systemctl restart mattermost.service it outputs the following:

Job for mattermost.service failed because the control process exited with error code.

journalctl -xeu mattermost.service outputs the following:

Apr 25 07:31:25 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:25.195 Z","level":"error","msg":"Failed to sync plugin from file store","caller":"app/plugin.go:344","plugin_id>
Apr 25 07:31:25 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:25.677 Z","level":"info","msg":"Installing extracted plugin","caller":"app/plugin_install.go:427","plugin_id":">
Apr 25 07:31:26 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:26.118 Z","level":"info","msg":"Installing extracted plugin","caller":"app/plugin_install.go:427","plugin_id":">
Apr 25 07:31:27 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:27.319 Z","level":"info","msg":"connectDatabase","caller":"app/plugin_api.go:1000","plugin_id":"focalboard","db>
Apr 25 07:31:27 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:27.334 Z","level":"info","msg":"Schema migration table is correct format","caller":"app/plugin_api.go:1000","pl>
Apr 25 07:31:27 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:27.334 Z","level":"info","msg":"{\"level\":\"info\",\"msg\":\"Pinging SQL\",\"fields\":{\"database\":\"master\">
Apr 25 07:31:27 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:27.384 Z","level":"info","msg":"Initialized notification backend","caller":"app/plugin_api.go:1000","plugin_id">
Apr 25 07:31:27 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:27.385 Z","level":"info","msg":"Initialized notification backend","caller":"app/plugin_api.go:1000","plugin_id">
Apr 25 07:31:27 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:27.385 Z","level":"info","msg":"Initialized notification backend","caller":"app/plugin_api.go:1000","plugin_id">
Apr 25 07:31:27 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:27.399 Z","level":"info","msg":"Focalboard server","caller":"app/plugin_api.go:1000","plugin_id":"focalboard",">
Apr 25 07:31:27 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:27.399 Z","level":"info","msg":"Server.Start","caller":"app/plugin_api.go:1000","plugin_id":"focalboard"}
Apr 25 07:31:27 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:27.409 Z","level":"info","msg":"Boards product successfully started.","caller":"app/plugin_api.go:1000","plugin>
Apr 25 07:31:27 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:27.421 Z","level":"info","msg":"Processing prepackaged plugin","caller":"app/plugin.go:972","bundle_path":"/opt>
Apr 25 07:31:27 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:27.421 Z","level":"info","msg":"Processing prepackaged plugin","caller":"app/plugin.go:972","bundle_path":"/opt>
Apr 25 07:31:27 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:27.421 Z","level":"info","msg":"Processing prepackaged plugin","caller":"app/plugin.go:972","bundle_path":"/opt>
Apr 25 07:31:27 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:27.422 Z","level":"info","msg":"Processing prepackaged plugin","caller":"app/plugin.go:972","bundle_path":"/opt>
Apr 25 07:31:27 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:27.424 Z","level":"info","msg":"Processing prepackaged plugin","caller":"app/plugin.go:972","bundle_path":"/opt>
Apr 25 07:31:27 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:27.434 Z","level":"info","msg":"Processing prepackaged plugin","caller":"app/plugin.go:972","bundle_path":"/opt>
Apr 25 07:31:27 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:27.449 Z","level":"info","msg":"Processing prepackaged plugin","caller":"app/plugin.go:972","bundle_path":"/opt>
Apr 25 07:31:27 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:27.464 Z","level":"info","msg":"Processing prepackaged plugin","caller":"app/plugin.go:972","bundle_path":"/opt>
Apr 25 07:31:27 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:27.478 Z","level":"info","msg":"Processing prepackaged plugin","caller":"app/plugin.go:972","bundle_path":"/opt>
Apr 25 07:31:27 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:27.884 Z","level":"info","msg":"Not installing prepackaged plugin: not previously enabled","caller":"app/plugin>
Apr 25 07:31:27 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:27.931 Z","level":"info","msg":"Not installing prepackaged plugin: not previously enabled","caller":"app/plugin>
Apr 25 07:31:28 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:27.989 Z","level":"info","msg":"Installing extracted plugin","caller":"app/plugin_install.go:427","plugin_id":">
Apr 25 07:31:28 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:27.991 Z","level":"warn","msg":"Skipping local installation of plugin since not a newer version","caller":"app/>
Apr 25 07:31:28 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:28.013 Z","level":"warn","msg":"Error loading local plugin icon","caller":"app/plugin.go:1184","plugin_id":"mat>
Apr 25 07:31:28 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:28.013 Z","level":"info","msg":"Not installing prepackaged plugin: not previously enabled","caller":"app/plugin>
Apr 25 07:31:28 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:28.055 Z","level":"info","msg":"Not installing prepackaged plugin: not previously enabled","caller":"app/plugin>
Apr 25 07:31:28 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:28.064 Z","level":"info","msg":"Not installing prepackaged plugin: not previously enabled","caller":"app/plugin>
Apr 25 07:31:28 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:28.101 Z","level":"info","msg":"Not installing prepackaged plugin: not previously enabled","caller":"app/plugin>
Apr 25 07:31:28 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:28.140 Z","level":"info","msg":"Installing extracted plugin","caller":"app/plugin_install.go:427","plugin_id":">
Apr 25 07:31:28 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:28.141 Z","level":"warn","msg":"Skipping local installation of plugin since not a newer version","caller":"app/>
Apr 25 07:31:28 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:28.269 Z","level":"info","msg":"Installing extracted plugin","caller":"app/plugin_install.go:427","plugin_id":">
Apr 25 07:31:28 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:28.278 Z","level":"error","msg":"Failed to install prepackaged plugin","caller":"app/plugin.go:939","bundle_pat>
Apr 25 07:31:28 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:28.280 Z","level":"error","msg":"Unable to activate plugin","caller":"app/plugin.go:144","plugin_id":"com.matte>
Apr 25 07:31:28 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:28.280 Z","level":"error","msg":"Unable to activate plugin","caller":"app/plugin.go:144","plugin_id":"com.matte>
Apr 25 07:31:28 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:28.356 Z","level":"info","msg":"Starting Server...","caller":"app/server.go:877"}
Apr 25 07:31:28 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:28.356 Z","level":"error","msg":"Error starting server, err:listen tcp :443: bind: permission denied: listen tc>
Apr 25 07:31:28 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:28.356 Z","level":"info","msg":"Stopping Server...","caller":"app/server.go:659"}
Apr 25 07:31:28 ov-9a01db mattermost[33595]: {"timestamp":"2024-04-25 07:31:28.522 Z","level":"info","msg":"Shutting down Email batching service...","caller":"email/service.go:85"}

I don’t really know what else I can try. Any help or suggestions most welcome !

2

Never mind ! I just found the solution ! :sweat_smile:
I had to activate again the CAP_NET_BIND_SERVICE capability as indicated on Configure TLS on Mattermost server - Mattermost documentation

2 Likes
3

I love this… so many times I’ve beaten my head against a problem, posted… and 30 seconds later was like OH YEAH, that’s what it is! :smiley: Thanks for posting your resolution.

1 Like