Cannot get mattermost service started

Troubleshooting
1

Summary
A new installation on AWS with MySQL RDS, mattermost.service does not start

Steps to reproduce
This is my first time setting up Mattermost. I’ve never installed or ran it before, so I may have made every mistake in the book and some you’ve never heard of.
I’m attempting to install mattermost on a AWS ec2 and use AWS RDS MySQL database.
I’ve followed every step in Install Mattermost Server using the tarball — Mattermost documentation
Failure occurs when attempting to run
sudo systemctl start mattermost

Expected behavior
Mattermost runs.

Observed behavior
bitnami@ip-172-31-46-220:~$ sudo systemctl start mattermost
Job for mattermost.service failed because the control process exited with error code.
See “systemctl status mattermost.service” and “journalctl -xe” for details.
bitnami@ip-172-31-46-220:~$ systemctl status mattermost.service
● mattermost.service - Mattermost
Loaded: loaded (/lib/systemd/system/mattermost.service; disabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Wed 2024-02-21 22:42:24 UTC; 347ms ago
Process: 20827 ExecStart=/opt/mattermost/bin/mattermost (code=exited, status=1/FAILURE)
Main PID: 20827 (code=exited, status=1/FAILURE)
CPU: 338ms

Feb 21 22:42:24 ip-172-31-46-220 systemd[1]: mattermost.service: Main process exited, code=exited, status=1/FAILURE
Feb 21 22:42:24 ip-172-31-46-220 systemd[1]: mattermost.service: Failed with result ‘exit-code’.
Feb 21 22:42:24 ip-172-31-46-220 systemd[1]: Failed to start Mattermost.

bitnami@ip-172-31-46-220:~$ journalctl -xe
Feb 21 22:45:12 ip-172-31-46-220 mattermost[20912]: export Export data from Mattermost
Feb 21 22:45:12 ip-172-31-46-220 mattermost[20912]: help Help about any command
Feb 21 22:45:12 ip-172-31-46-220 mattermost[20912]: import Import data.
Feb 21 22:45:12 ip-172-31-46-220 mattermost[20912]: jobserver Start the Mattermost job server
Feb 21 22:45:12 ip-172-31-46-220 mattermost[20912]: server Run the Mattermost server
Feb 21 22:45:12 ip-172-31-46-220 mattermost[20912]: version Display version information
Feb 21 22:45:12 ip-172-31-46-220 mattermost[20912]: Flags:
Feb 21 22:45:12 ip-172-31-46-220 mattermost[20912]: -c, --config string Configuration file to use.
Feb 21 22:45:12 ip-172-31-46-220 mattermost[20912]: -h, --help help for mattermost
Feb 21 22:45:12 ip-172-31-46-220 mattermost[20912]: Use “mattermost [command] --help” for more information about a command.
Feb 21 22:45:12 ip-172-31-46-220 systemd[1]: mattermost.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: Debian -- User Support
░░
░░ An ExecStart= process belonging to unit mattermost.service has exited.
░░
░░ The process’ exit code is ‘exited’ and its exit status is 1.
Feb 21 22:45:12 ip-172-31-46-220 systemd[1]: mattermost.service: Failed with result ‘exit-code’.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: Debian -- User Support
░░
░░ The unit mattermost.service has entered the ‘failed’ state with result ‘exit-code’.
Feb 21 22:45:12 ip-172-31-46-220 systemd[1]: Failed to start Mattermost.
░░ Subject: A start job for unit mattermost.service has failed
░░ Defined-By: systemd
░░ Support: Debian -- User Support
░░
░░ A start job for unit mattermost.service has finished with a failure.
░░
░░ The job identifier is 213759 and the job result is failed.

I believe the content of my mattermost.service file may be to blame, but I can’t find the culprit

bitnami@ip-172-31-46-220:~$ more /lib/systemd/system/mattermost.service
[Unit]
Description=Mattermost
After=network.target

[Service]
Type=notify
ExecStart=/opt/mattermost/bin/mattermost
TimeoutStartSec=3600
KillMode=mixed
Restart=always
RestartSec=10
WorkingDirectory=/opt/mattermost
User=mattermost
Group=mattermost
LimitNOFILE=49152

[Install]
WantedBy=multi-user.target

and here are the edits I made to config.json.
Under Service Settings, I set SiteURL this way
“ServiceSettings”: {
“SiteURL”: “http://172.31.46.220”,

Under SqlSettings I made these edits
“SqlSettings”: {
“DriverName”: “mysql”,
“DataSource”: “mysql://admin:<*******>@chatdatabase1.c5uemsg8swm9.us-east-2.rds.amazonaws.com:3306/chatdatabase1?sslmode=disable\u0026connect_timeout=10”,

These are all the edits I made. Please ask me anything you want.

Thank you for your help.

2

First things that I’d check given what you’ve shown.

  • Does the user mattermost exist
  • Does the group mattermost exists and is the user a part of it
  • Are /opt/mattermost and underlying the underlying directories permissioned for the mattermost user/group to read / execute.
  • What do I see in /opt/mattermost/log/mattermost.log ?
3

  • Does the user mattermost exist? Does the group mattermost exists and is the user a part of it?
    Yes
    bitnami@ip-172-31-46-220:~$ sudo useradd --system --user-group mattermost
    useradd: user ‘mattermost’ already exists
    bitnami@ip-172-31-46-220:~$

  • Are /opt/mattermost and underlying the underlying directories permissioned for the mattermost user/group to read / execute.
    drwxrwxr-x 11 mattermost mattermost 4096 Feb 21 03:33 mattermost
    bitnami@ip-172-31-46-220:~$ ls -l /opt/mattermost/
    total 668
    drwxrwxr-x 2 mattermost mattermost 4096 Feb 12 21:17 bin
    drwxrwxr-x 7 mattermost mattermost 20480 Feb 12 21:16 client
    drwxrwxr-x 2 mattermost mattermost 4096 Feb 22 02:42 config
    drwxrwxr-x 2 mattermost mattermost 4096 Feb 21 03:33 data
    -rw-rw-r-- 1 mattermost mattermost 2046 Feb 12 21:16 ENTERPRISE-EDITION-LICENSE.txt
    drwxrwxr-x 2 mattermost mattermost 4096 Feb 12 21:16 fonts
    drwxrwxr-x 2 mattermost mattermost 4096 Feb 12 21:16 i18n
    drwxrwxr-x 2 mattermost mattermost 4096 Feb 21 12:31 logs
    -rw-rw-r-- 1 mattermost mattermost 448 Feb 12 21:16 manifest.txt
    -rw-rw-r-- 1 mattermost mattermost 611152 Feb 12 21:16 NOTICE.txt
    drwxrwxr-x 2 mattermost mattermost 4096 Feb 12 21:17 prepackaged_plugins
    -rw-rw-r-- 1 mattermost mattermost 7528 Feb 12 21:16 README.md
    drwxrwxr-x 2 mattermost mattermost 4096 Feb 12 21:16 templates

I can show more if you want.

  • What do I see in /opt/mattermost/logs/mattermost.log ?
    Here are the last lines in from today. Didn’t check this previously. Seeing a pattern, but not seeing specifically what the issue is. I’m guessing a connectoin to my mysql database is not working butnot sure what to correct yet.

{“timestamp”:“2024-02-22 20:12:41.068 Z”,“level”:“info”,“msg”:“Server is initializing…”,“caller”:“platform/service.go:173”,“go_version”:“go1.20.7”}
{“timestamp”:“2024-02-22 20:12:41.070 Z”,“level”:“error”,“msg”:“failed to initialize platform: cannot create store: error setting up connections: failed to reset read timeout from datasource: default addr for network ‘chatdatabase1.c5uemsg8swm9.us-east-2.rds.amazonaws.com:3306’ unknown”,“caller”:“commands/server.go:76”}
{“timestamp”:“2024-02-22 21:02:54.339 Z”,“level”:“info”,“msg”:“Server is initializing…”,“caller”:“platform/service.go:173”,“go_version”:“go1.20.7”}
{“timestamp”:“2024-02-22 21:13:45.333 Z”,“level”:“info”,“msg”:“Server is initializing…”,“caller”:“platform/service.go:173”,“go_version”:“go1.20.7”}
{“timestamp”:“2024-02-22 21:13:45.334 Z”,“level”:“error”,“msg”:“failed to initialize platform: cannot create store: error setting up connections: failed to reset read timeout from datasource: default addr for network ‘chatdatabase1.c5uemsg8swm9.us-east-2.rds.amazonaws.com:3306’ unknown”,“caller”:“commands/server.go:76”}
{“timestamp”:“2024-02-22 21:14:06.339 Z”,“level”:“info”,“msg”:“Server is initializing…”,“caller”:“platform/service.go:173”,“go_version”:“go1.20.7”}
{“timestamp”:“2024-02-22 21:14:06.340 Z”,“level”:“error”,“msg”:“failed to initialize platform: cannot create store: error setting up connections: failed to reset read timeout from datasource: default addr for network ‘chatdatabase1.c5uemsg8swm9.us-east-2.rds.amazonaws.com:3306’ unknown”,“caller”:“commands/server.go:76”}
{“timestamp”:“2024-02-22 22:03:59.068 Z”,“level”:“info”,“msg”:“Server is initializing…”,“caller”:“platform/service.go:173”,“go_version”:“go1.20.7”}
{“timestamp”:“2024-02-22 22:03:59.069 Z”,“level”:“error”,“msg”:“failed to initialize platform: cannot create store: error setting up connections: failed to reset read timeout from datasource: default addr for network ‘chatdatabase1.c5uemsg8swm9.us-east-2.rds.amazonaws.com:3306’ unknown”,“caller”:“commands/server.go:76”}
{“timestamp”:“2024-02-22 22:05:33.559 Z”,“level”:“info”,“msg”:“Server is initializing…”,“caller”:“platform/service.go:173”,“go_version”:“go1.20.7”}
{“timestamp”:“2024-02-22 22:05:33.560 Z”,“level”:“error”,“msg”:“failed to initialize platform: cannot create store: error setting up connections: failed to reset read timeout from datasource: default addr for network ‘chatdatabase1.c5uemsg8swm9.us-east-2.rds.amazonaws.com:3306’ unknown”,“caller”:“commands/server.go:76”}
{“timestamp”:“2024-02-22 22:05:54.579 Z”,“level”:“info”,“msg”:“Server is initializing…”,“caller”:“platform/service.go:173”,“go_version”:“go1.20.7”}
{“timestamp”:“2024-02-22 22:05:54.579 Z”,“level”:“error”,“msg”:“failed to initialize platform: cannot create store: error setting up connections: failed to reset read timeout from datasource: default addr for network ‘chatdatabase1.c5uemsg8swm9.us-east-2.rds.amazonaws.com:3306’ unknown”,“caller”:“commands/server.go:76”}

4

What version of Mattermost are you using? And what version of MySQL? We’ve recently dropped support for MySQL 5.7, and will be deprecating support for MySQL entirely soon. Take a look at Mattermost changelog — Mattermost documentation and Migration guidelines from MySQL to PostgreSQL — Mattermost documentation

Can you log in to your database with the mysql command and the credentials / destination in DataSource from the Mattermost server?

What address does chatdatabase1.c5uemsg8swm9.us-east-2.rds.amazonaws.com resolve to for you? The error suggests that there’s a Layer 3 issue. What does your routing table look like?

Also, look at security group settings / firewall for the database instance.