Today I did an apt update and apt upgrade. After that Mattermost stopped working.
The version is Mattermost Omnibus. I think I isolated the issue to a failure of the postgresql database.
mmomni status gives me the following output:
[1736] nginx: running
[0] postgresql@12-main: dead
[2022] mattermost: running
I restarted the server and all services several times, which didn’t bring any change.
What Mattermost version did you upgrade to? What is your Postgres version? Are there any other logs you can help share?
edit 2:
mmomni status gives me the output
[0] postgresql@12-main: dead
But the installed instance of postgresql is version 13 instead of 12. Can this be the problem?
edit: the version is 5.29.1-0 and now I can install another upgrade to 5.30.1-0.
This is the output of the mattermost upgrade:
config file “/etc/mattermost/mmomni.yml” successfully saved
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match ‘all’
PLAY [localhost] ***************************************************************
TASK [Gathering Facts] *********************************************************
ok: [localhost]
TASK [Update debconf variables with json file values] **************************
ok: [localhost] => (item={‘key’: ‘mattermost-omnibus/domain’, ‘value’: ‘xxx’})
ok: [localhost] => (item={‘key’: ‘mattermost-omnibus/email’, ‘value’: 'xxx})
TASK [Configure NGINX http template] *******************************************
skipping: [localhost]
TASK [Ensure that NGINX is running] ********************************************
ok: [localhost]
TASK [Generate SSL Certificate] ************************************************
changed: [localhost]
TASK [Configure NGINX https template] ******************************************
ok: [localhost]
TASK [Delete default NGINX configuration file] *********************************
ok: [localhost]
TASK [Restart NGINX service with the new configuration] ************************
changed: [localhost]
TASK [Create database user] ****************************************************
[WARNING]: Module did not set no_log for no_password_changes
ok: [localhost]
TASK [Create database for Mattermost] ******************************************
ok: [localhost]
TASK [Create system user] ******************************************************
ok: [localhost]
TASK [Ensure file permissions] *************************************************
changed: [localhost] => (item=/opt/mattermost)
ok: [localhost] => (item=/var/opt/mattermost/data)
TASK [Generate mattermost env variables] ***************************************
ok: [localhost]
TASK [Generate systemd service] ************************************************
changed: [localhost]
TASK [Enable and restart Mattermost service] ***********************************
changed: [localhost]
PLAY RECAP *********************************************************************
localhost : ok=14 changed=5 unreachable=0 failed=0 skipped=1 rescued=0 igno red=0
And there was another error message when I run apt update:
Skipping acquire of configured file ‘nginx/binary-i386/Packages’ as repositor y ‘https:// nginx .org/packages/ubuntu focal InRelease’ doesn’t support architectu re ‘i386’
N: Skipping acquire of configured file ‘main/binary-i386/Packages’ as repository ‘http:// apt. postgresql .org/pub/repos/apt focal-pgdg InRelease’ doesn’t support architecture ‘i386’
N: Skipping acquire of configured file ‘main/binary-i386/Packages’ as repository ‘https:/ /deb. packages.mattermost .com focal InRelease’ doesn’t support architect ure ‘i386’
The Server is a VPS with x86 architecture .
This is the output of postgresql-13-main.log
2020-12-21 15:36:23.405 CET [663] LOG: starting PostgreSQL 13.1 (Ubuntu 13.1-1.pgdg20.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, 64-bit
2020-12-21 15:36:23.408 CET [663] LOG: listening on IPv6 address “::1”, port 5432
2020-12-21 15:36:23.408 CET [663] LOG: listening on IPv4 address “127.0.0.1”, port 5432
2020-12-21 15:36:23.411 CET [663] LOG: listening on Unix socket “/var/run/postgresql/.s.PGSQL.5432”
2020-12-21 15:36:23.422 CET [677] LOG: database system was shut down at 2020-12-21 15:36:06 CET
2020-12-21 15:36:23.621 CET [663] LOG: database system is ready to accept connections
2020-12-21 15:36:26.735 CET [744] mmuser@mattermost ERROR: relation “idx_teams_description” does not exist
2020-12-21 15:36:26.735 CET [744] mmuser@mattermost STATEMENT: SELECT $1::regclass
2020-12-21 17:00:36.273 CET [2029] mmuser@mattermost ERROR: relation “idx_teams_description” does not exist
2020-12-21 17:00:36.273 CET [2029] mmuser@mattermost STATEMENT: SELECT $1::regclass
2020-12-21 17:05:07.125 CET [2162] user@user FATAL: role “user” does not exist
2020-12-21 17:07:30.171 CET [2198] mattermost@mattermost FATAL: role “mattermost” does not exist
2020-12-21 17:18:29.536 CET [663] LOG: received SIGHUP, reloading configuration files
2020-12-21 17:22:02.662 CET [2463] mmuser@mattermost LOG: provided user name (mmuser) and authenticated user name (user) do not match
And the output of mattermost.log
{“level”:“info”,“ts”:1608558644.7119298,“caller”:“app/server.go:214”,“msg”:“Server is initializing…”}
{“level”:“info”,“ts”:1608558644.7136028,“caller”:“app/web_hub.go:83”,“msg”:“Starting websocket hubs”,“number_of_hubs”:2}
{“level”:“warn”,“ts”:1608558644.7139864,“caller”:“app/server.go:223”,“msg”:“Sentry reporting is enabled, but SENTRY_DSN is not set. Disabling reporting.”}
{“level”:“info”,“ts”:1608558644.7644162,“caller”:“utils/i18n.go:90”,“msg”:“Loaded system translations”,“for locale”:“en”,“from locale”:“/opt/mattermost/i18n/en.json”}
{“level”:“info”,“ts”:1608558644.77108,“caller”:“sqlstore/supplier.go:235”,“msg”:“Pinging SQL”,“database”:“master”}
{“level”:“error”,“ts”:1608558644.7780142,“caller”:“sqlstore/supplier.go:247”,“msg”:“Failed to ping DB”,“error”:“dial tcp [::1]:3306: connect: connection refused”,“retrying in seconds”:10}
{“level”:“info”,“ts”:1608558654.7791674,“caller”:“sqlstore/supplier.go:235”,“msg”:“Pinging SQL”,“database”:“master”}
{“level”:“error”,“ts”:1608558654.7817082,“caller”:“sqlstore/supplier.go:247”,“msg”:“Failed to ping DB”,“error”:“dial tcp [::1]:3306: connect: connection refused”,“retrying in seconds”:10}
{“level”:“info”,“ts”:1608558664.78197,“caller”:“sqlstore/supplier.go:235”,“msg”:“Pinging SQL”,“database”:“master”}
{“level”:“error”,“ts”:1608558664.7832212,“caller”:“sqlstore/supplier.go:247”,“msg”:“Failed to ping DB”,“error”:“dial tcp [::1]:3306: connect: connection refused”,“retrying in seconds”:10}
{“level”:“info”,“ts”:1608558717.2512,“caller”:“app/server.go:214”,“msg”:“Server is initializing…”}
{“level”:“info”,“ts”:1608558717.2515335,“caller”:“app/web_hub.go:83”,“msg”:“Starting websocket hubs”,“number_of_hubs”:2}
{“level”:“warn”,“ts”:1608558717.2517383,“caller”:“app/server.go:223”,“msg”:“Sentry reporting is enabled, but SENTRY_DSN is not set. Disabling reporting.”}
{“level”:“info”,“ts”:1608558717.2872458,“caller”:“utils/i18n.go:90”,“msg”:“Loaded system translations”,“for locale”:“en”,“from locale”:“/opt/mattermost/i18n/en.json”}
{“level”:“info”,“ts”:1608558717.3002326,“caller”:“sqlstore/supplier.go:235”,“msg”:“Pinging SQL”,“database”:“master”}
{“level”:“error”,“ts”:1608558717.3271334,“caller”:“sqlstore/supplier.go:247”,“msg”:“Failed to ping DB”,“error”:“dial tcp [::1]:3306: connect: connection refused”,“retrying in seconds”:10}
Upgrading a DB major version can cause this. Most probably you may need to manually run pg_upgrade in order to convert the data to the new version. More info at https://www.postgresql.org/docs/current/upgrading.html