Self-hosted server service won't start

1

When trying to start the mattermost service on a self-hosted server running Ubuntu 24.04, I get:
manager@raspberrypi:~$ sudo systemctl restart mattermost.service
Job for mattermost.service failed because the control process exited with error code.
See “systemctl status mattermost.service” and “journalctl -xeu mattermost.service” for details.
manager@raspberrypi:~$ sudo systemctl status mattermost.service
● mattermost.service - Mattermost
Loaded: loaded (/usr/lib/systemd/system/mattermost.service; disabled; pres>
Active: activating (auto-restart) (Result: exit-code) since Thu 2025-07-31>
Process: 113140 ExecStart=/opt/mattermost/bin/mattermost (code=exited, stat>
Main PID: 113140 (code=exited, status=203/EXEC)
CPU: 17ms

My box already runs nginx successfully. Two changes I made I was unclear about was these in in /opt/mattermost/config/config.json:
SQLSettings:
“DataSource”: “postgres://mmuser:*******@192.168.2.10:5432/mattermost?sslmode=disable&connect_timeout=10”,
where 192.168.2.10 is the NAT address of the box and ******** is the Postgres password
and ServiceSetting:
in "SiteURL": The domain name for the Mattermost application, where I used johnrose.mywire.org which is my web server’s address on that box.
Any ideas as to what steps to take to ascertain the cause(s) of the problem?

2

Seems you cut off the relevant part with the log messages.

Can you instead please post the output of

journalctl -xeu mattermost

What exactly do you mean with “NAT address” and what is “the box”? So is your server reachable from the Internet or just internal?

3

NAT address is the internal address of my server on my home network, which is statically assigned by my router whose NAT address is 192.168.2.1 i.e. my server is on the same network as my router which assigned a DHCP address to my server. Box is another word for PC or computer: in this case it refers to my server which runs Ubuntu 24.04 LTS and has postgresql & mattermost as systemd services. My server is reachable from the internet e.g. https://johnrose.mywire.org/summer brings back a photo of my cat.

I have run ‘journalctl -xeu mattermost’ but it gives 6,000 lines. Here is perhaps the required relevant part:
manager@raspberrypi:~$ journalctl -xeu mattermost
Jul 31 14:27:48 raspberrypi (ttermost)[136495]: mattermost.service: Failed to e>
Jul 31 14:27:48 raspberrypi systemd[1]: mattermost.service: Main process exited>
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ An ExecStart= process belonging to unit mattermost.service has exited.
░░
░░ The process’ exit code is ‘exited’ and its exit status is 203.
Jul 31 14:27:48 raspberrypi systemd[1]: mattermost.service: Failed with result >
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: Enterprise open source support | Ubuntu
░░
░░ The unit mattermost.service has entered the ‘failed’ state with result 'exit>
Jul 31 14:27:48 raspberrypi systemd[1]: Failed to start mattermost.service - Ma>
░░ Subject: A start job for unit mattermost.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit mattermost.service has finished with a failure.
░░
░░ The job identifier is 412247 and the job result is failed.
lines 6309-6331/6331 (END)

4

Hm, the output indicates that it is cut off (because of the “>” sign which normally shows that there is more text if you press the “→” key on your keyboard).

Maybe this one works:

journalctl -xeu mattermost | tail -n 50 | tee -a ~/20250731_mattermost_troubleshooting.txt

“tail -n 50” prints the last 50 lines only and “tee -a” writes/appends the output to a file, but also prints it to STDOUT so that you can see it on your shell, too. Hopefully there is more information about the crash. If you don’t have “tee” installed, just use “…tail -n 50 >> ~/20250731….”, that redirects STDOUT to the file (without showing it on your shell)

Is there a log file in the mattermost installation directory on your server? Should be “/opt/mattermost/logs” if you followed the installation instructions.

5

Link to the journalctl output is https://app.box.com/s/z9msaz3ykchy2r5wh63ftsau5fwl2k5p

No log file in /opt/mattermost/logs or mattermost in /var/log.

6

Ok, so the main error seems to be:

Jul 31 15:28:16 raspberrypi (ttermost)[141734]: mattermost.service: Failed to execute /opt/mattermost/bin/mattermost: Exec format error
Jul 31 15:28:16 raspberrypi systemd[1]: mattermost.service: Main process exited, code=exited, status=203/EXEC

According to troubleshooting guides this error could be caused by “running binaries compiled for a different architecture” (i.e. program compiled for “x64” on an “x86” system, “ARM”-based executable on an x86-based system, …). It’s also possible that the binary is corrupt or the file format is invalid/incompatible.

You may try the following commands:

# show content of the folder incl. permissions etc.
ls -lah /opt/mattermost/bin/
ls -lah /opt/mattermost/
ls -lah /opt/

# check the architecture of the file
file /opt/mattermost/bin/mattermost

# check the architecture of your server
uname -m # show architecture only
uname -a # show detailed information

# show systemd unit file for mattermost.service
cat /usr/lib/systemd/system/mattermost.service

Apart from that, how did you install mattermost on your ubuntu server in the first place?

7

My systemd unit file for mattermost.service looks like this:

[Unit]
Description=Mattermost
After=syslog.target network.target

[Service]
StandardOutput=null
StandardError=null
Type=notify
ExecStart=/opt/mattermost/bin/mattermost
TimeoutStartSec=3600
KillMode=mixed
Restart=always
RestartSec=10
WorkingDirectory=/opt/mattermost
User=mattermost
PIDFile=/var/spool/mattermost/pid/master.pid
LimitNOFILE=49152
Environment=HTTP_PROXY=<Proxy>
Environment=HTTPS_PROXY=<Proxy>
Environment=NO_PROXY=<some URLs>

[Install]
WantedBy=multi-user.target
8

I didn’t fully read the instructions in https://docs.mattermost.com/deploy/server/deploy-linux.html and didn’t notice the sentence about arm64. So I downloaded the amd64 rather than the arm64 tarball! I’ve now removed mattermost and followed the instructions again with the arm64 tarball. Now getting a problem with Postgres when sserver.

manager@raspberrypi:~$ sudo systemctl start mattermost
[sudo] password for manager:
Job for mattermost.service failed because the control process exited with error code.
See “systemctl status mattermost.service” and “journalctl -xeu mattermost.service” for details.
manager@raspberrypi:~$ journalctl -xeu mattermost | tail -n 50 | tee -a ~/20250731_mattermost_troubleshooting.txt
Aug 01 06:53:15 raspberrypi mattermost[10357]: {“timestamp”:“2025-08-01 06:53:15.738 +01:00”,“level”:“info”,“msg”:“Pinging SQL”,“caller”:“sql/sql_utils.go:68”,“database”:“master”,“dataSource”:“postgres://:@192.168.2.10:5432/mattermost?connect_timeout=10&sslmode=disable”,“attempt”:3}
Aug 01 06:53:15 raspberrypi mattermost[10357]: {“timestamp”:“2025-08-01 06:53:15.779 +01:00”,“level”:“error”,“msg”:“Failed to ping DB”,“caller”:“sql/sql_utils.go:77”,“database”:“master”,“dataSource”:“postgres://:@192.168.2.10:5432/mattermost?connect_timeout=10&sslmode=disable”,“retrying in seconds”:2,“error”:“pq: password authentication failed for user “mmuser””}
Aug 01 06:53:17 raspberrypi mattermost[10357]: {“timestamp”:“2025-08-01 06:53:17.780 +01:00”,“level”:“info”,“msg”:“Pinging SQL”,“caller”:“sql/sql_utils.go:68”,“database”:“master”,“dataSource”:“postgres://:@192.168.2.10:5432/mattermost?connect_timeout=10&sslmode=disable”,“attempt”:4}
Aug 01 06:53:17 raspberrypi mattermost[10357]: {“timestamp”:“2025-08-01 06:53:17.814 +01:00”,“level”:“error”,“msg”:“Failed to ping DB”,“caller”:“sql/sql_utils.go:77”,“database”:“master”,“dataSource”:“postgres://:@192.168.2.10:5432/mattermost?connect_timeout=10&sslmode=disable”,“retrying in seconds”:2,“error”:“pq: password authentication failed for user “mmuser””}
Aug 01 06:53:19 raspberrypi mattermost[10357]: {“timestamp”:“2025-08-01 06:53:19.815 +01:00”,“level”:“info”,“msg”:“Pinging SQL”,“caller”:“sql/sql_utils.go:68”,“database”:“master”,“dataSource”:“postgres://:@192.168.2.10:5432/mattermost?connect_timeout=10&sslmode=disable”,“attempt”:5}
Aug 01 06:53:19 raspberrypi mattermost[10357]: Error: failed to initialize platform: cannot create store: error setting up connections: pq: password authentication failed for user “mmuser”
Aug 01 06:53:19 raspberrypi mattermost[10357]: Usage:
Aug 01 06:53:19 raspberrypi mattermost[10357]: mattermost [flags]
Aug 01 06:53:19 raspberrypi mattermost[10357]: mattermost [command]
Aug 01 06:53:19 raspberrypi mattermost[10357]: Available Commands:
Aug 01 06:53:19 raspberrypi mattermost[10357]: completion Generate the autocompletion script for the specified shell
Aug 01 06:53:19 raspberrypi mattermost[10357]: db Commands related to the database
Aug 01 06:53:19 raspberrypi mattermost[10357]: export Export data from Mattermost
Aug 01 06:53:19 raspberrypi mattermost[10357]: help Help about any command
Aug 01 06:53:19 raspberrypi mattermost[10357]: import Import data.
Aug 01 06:53:19 raspberrypi mattermost[10357]: jobserver Start the Mattermost job server
Aug 01 06:53:19 raspberrypi mattermost[10357]: server Run the Mattermost server
Aug 01 06:53:19 raspberrypi mattermost[10357]: version Display version information
Aug 01 06:53:19 raspberrypi mattermost[10357]: Flags:
Aug 01 06:53:19 raspberrypi mattermost[10357]: -c, --config string Configuration file to use.
Aug 01 06:53:19 raspberrypi mattermost[10357]: -h, --help help for mattermost
Aug 01 06:53:19 raspberrypi mattermost[10357]: Use “mattermost [command] --help” for more information about a command.
Aug 01 06:53:19 raspberrypi systemd[1]: mattermost.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ An ExecStart= process belonging to unit mattermost.service has exited.
░░
░░ The process’ exit code is ‘exited’ and its exit status is 1.
Aug 01 06:53:19 raspberrypi systemd[1]: mattermost.service: Failed with result ‘exit-code’.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit mattermost.service has entered the ‘failed’ state with result ‘exit-code’.
Aug 01 06:53:19 raspberrypi systemd[1]: Failed to start mattermost.service - Mattermost.
░░ Subject: A start job for unit mattermost.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit mattermost.service has finished with a failure.
░░
░░ The job identifier is 4116 and the job result is failed.
Aug 01 06:53:19 raspberrypi systemd[1]: mattermost.service: Consumed 1.362s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit mattermost.service completed and consumed the indicated resources.

Could this be due to my router having my server as the DMZ but not yet having iptables set to accept the numerous ports for mattermost?

PS I use TalkTalk which doesn’t allow my own website https://johnrose.mywire.org to be accessed by my internet browser on my home network. Could that be a problem?

9

The error message “pq: password authentication failed for user “mmuser”” is pretty clear imho. How did you install postgres and where? Is it initialized, yet?

Please try to login on your mattermost database with the postgres shell tools and verify that the used password is correct