Mattermost blocks on pinging database

Troubleshooting
1

Summary

Mattermost do not go further than “Pinging SQL master database” on startup.

Steps to reproduce

Mattermost team 5.12.3 3.13.1
Ubuntu 18.04
Postgresql 10

config.json :
“SqlSettings”: {
“DriverName”: “postgres”,
“DataSource”: “postgres://mmuser:<my_password>@localhost:5432/mattermost?sslmode=disable&connect_timeout=10”,

I’ve tested db connection with :

psql --dbname=mattermost --username=mmuser --password

I’ve got access to db without any problem

Then I tried to start the server with :

sudo -u mattermost ./bin/mattermost

The last message is “Pinging SQL master database”

Here are the last logs for postgresql :

2019-07-23 09:27:06.537 CEST [17929] mmuser@mattermost ERROR: relation “idx_teams_description” does not exist
2019-07-23 09:27:06.537 CEST [17929] mmuser@mattermost STATEMENT: SELECT $1::regclass

2019-07-22 14:03:39.143 CEST [19023] mmuser@mattermost ERROR: duplicate key value violates unique constraint “users_username_key”
2019-07-22 14:03:39.143 CEST [19023] mmuser@mattermost DETAIL: Key (username)=(surveybot) already exists.
2019-07-22 14:03:39.143 CEST [19023] mmuser@mattermost STATEMENT:
insert into “users” (“id”,“createat”,“updateat”,“deleteat”,“username”,“password”,“authdata”,“authservice”,“email”,
“emailverified”,“nickname”,“firstname”,“lastname”,“position”,“roles”,“allowmarketing”,“props”,“notifyprops”,
“lastpasswordupdate”,“lastpictureupdate”,“failedattempts”,“locale”,“timezone”,“mfaactive”,“mfasecret”) values
($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25);

Expected behavior

The server should start or I should get an error.

Observed behavior

My server’s disk got full, so some sevices stopped working. I’ve clean my disk and restart all services but mattermost does not start anymore.

Update

I’ve upgraded to version 5.13.1 and the issue remains.

2

Hi @VoyagesDivins, as a first step, wondering if some of these earlier threads might help:

3

Hi, thanks for replying.

I don’t think it’s a firewall problem as db and service are on the same server.

Also I do not get messages of connection failed. For instance if I intentionnaly make a mistake, say in port number, I immediately get a message

"Failed to ping DB retrying in 10 seconds

When the connection is correct, I don’t get any message.

Another thing that I don’t if revelant : in the config.json the “&” character in postgresql config gets converted to \u0026

I don’t have any special characters in my password.

4

Hi @VoyagesDivins, a response from a community member: If the interruption (full disk) occurred during install, it might be worth a try to drop the mattermost database and try again.

5

The problem didn’t occure during install. Server was already up and running.

As you suggest, the problem came from the database.

Table “jobs” had more than 15000 rows.

I did a hard truncate on that table and I was able to start mattermost server again.

Thank you

6

The problem occured again after a server reboot. This time truncating “jobs” table didn’t help.

7

The problem occured again when trying to migrate my data on another server, any ideas ? should I open a issue on github ?

8

@VoyagesDivins, sorry to hear about the trouble you keep running into. The behaviour you describe is expected from the Mattermost server: if it cannot communicate with your PostgresSQL instance, it will block on startup, eventually stopping altogether after a fixed number of attempts.

I’d recommend trying to open a manual psql connection to your database from the same server running the Mattermost server and continue your debugging from there.

9

From psql, running

SELECT pid, age(clock_timestamp(), query_start), usename, query
FROM pg_stat_activity
WHERE query != ‘’ AND query NOT ILIKE ‘%pg_stat_activity%’
ORDER BY query_start desc;

Two queries running under mattermost user are listed as follow :

10550 | 00:14:26.840856 | mattermost | SELECT * FROM Systems
9500 | 00:29:01.626238 | mattermost | SELECT * FROM Systems

Stopping mattermost and Killing queries with

SELECT pg_terminate_backend(10550);
SELECT pg_terminate_backend(9500);

Then running mattermost again, the same query (only one) runs and never ends

10

The only occurence of this request in the sources is in file supplier.go line 290.

The complete request is

SELECT Value FROM Systems WHERE Name=‘Version’

Executing this from psql gives me

 value
--------
 5.15.0
(1 row)