"Failed to ping DB" when trying to start mattermost server

markll80 · October 15, 2020, 9:42pm

Summary

Greetings! I am following this guide to set up mattermost(5.27.0) on a Microsoft Azure server with centos8. I was trying to use:

sudo -u mattermost ./bin/mattermost

to start the server but I was unable to ping the database. (Every previous steps in the guide worked fine).

My config.json:

"DriverName": "postgres",
"DataSource": "postgres://mmuser:mmuser_pw@20.48.100.154:8065/mattermost?sslmode=disable\u0026connect_timeout=10",

Expected behavior

Should be able to connect successfully.

Observed behavior

sudo -u mattermost ./bin/mattermost
{"level":"info","msg":"Server is initializing..."}
{"level":"info","ts":1602793035.114669,"caller":"app/web_hub.go:83","msg":"Starting websocket hubs","number_of_hubs":2}
{"level":"info","ts":1602793035.1163628,"caller":"utils/i18n.go:83","msg":"Loaded system translations","for locale":"en","from locale":"/opt/mattermost/i18n/en.json"}
{"level":"info","ts":1602793035.1180053,"caller":"sqlstore/supplier.go:227","msg":"Pinging SQL","database":"master"}
{"level":"error","ts":1602793045.1183646,"caller":"sqlstore/supplier.go:239","msg":"Failed to ping DB","error":"dial tcp 20.48.100.154:8065: i/o timeout","retrying in seconds":10}

XxLilBoPeepsxX · October 16, 2020, 3:11pm

Hi! Are you using a firewall such as UFW? If you are, could you verify that the port you are running your database on it open, and unrestricted and that your database user is configured correctly to connect to the Postgres database with the external IP?

In addition, are you connecting to a database that is located on the same server as your Mattermost instance, or is this a remote database?

markll80 · October 16, 2020, 8:36pm

Hello! Thanks for your response! To address your questions:

There is currently no firewall on the server.
The database and mattermost instance are located on the same server and I can connect to the postgres database with mmuser. Should I change the ip address to something else in my config.json?

Sorry if the question seem basic, I’m pretty new to this.

XxLilBoPeepsxX · October 18, 2020, 6:22pm

@markll80 I’m thinking this may be a networking error.

If you are hosting the Mattermost instance on the same server as the database, but you have the configured database user attempting to connect to the database as if it was connecting to a remote database on a different server (which is done by using the public IP address), then you may be having a routing conflict. This would be due to the Mattermost database connection agent attempting to connect to itself remotely, when instead, you should be using a database user that is configured to only be able to connect from the localhost.

For example, with MySQL, the database user that I am using, is mmuser@localhost. This means that the database user is directly connecting back to the local database, which is knows due to being an account connecting at the localhost, is on the local machine.

I hope this makes sense and helps, let me know if it works out, and if it doesn’t, I’ll continue to help you troubleshoot!

XxLilBoPeepsxX · October 18, 2020, 6:24pm

Also, I ran a nmap scan on the IP address you included in the sample code, and it doesn’t have any ports open, so could not possibly connect to the database even if everything else was setup correctly. Is the IP in the code example you initially provided the actual production IP address, or just a placeholder?

markll80 · October 19, 2020, 3:09pm

@XxLilBoPeepsxX
Yes changing the ip address to locahost and removing the port number works for me, thank you very much for your help!

XxLilBoPeepsxX · October 19, 2020, 3:21pm

I’m glad you got it working and glad to provide assistance, anytime!

KnifeHand · March 21, 2023, 7:41pm

Hello, I am having the same issue but I cannot resolve by changing the ip address to localhost and removing the port number.

I am using the docker-compose from this repo —> GitHub - mattermost/docker: Install Mattermost server via Docker
and launching it on a Ubuntu EC2 ami. The only thing I am editing is the env.example where I set the domain to my machine’s ip address and changing the image to the “team” edition.

I get the same error when Mattermost goes to ping the DB on postgres

`"timestamp":"2023-03-21 15:15:49.867 Z","level":"error","msg":"Failed to ping DB","caller":"sqlstore/store.go:250","error":"dial tcp 127.0.0.1:5432: connect: connection refused","retrying in seconds":10}`

This is the env var responsible for talking to the DB

`MM_SQLSETTINGS_DATASOURCE=postgres:${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}?sslmode=disable&connect_timeout=10`

I have replaced @postgres:5432 with the eth0 ip of the postgres container . I have tried the localhost address. I have also tried using @localhost:5432.

I don’t know what the magical incantation is and I don’t know why it runs perfectly fine on my local machine.

agriesser · March 22, 2023, 4:26am

Hi @KnifeHand and welcome to the Mattermost forums!

When using the docker-compose deployment, the postgres host should not be localhost because that would mean it’s running inside the Mattermost application container, which it doesn’t. It needs to be set to postgres (the defaultand this should also be the container name/alias of the database container you've spun up. So as long as your Mattermost containers both are running and if your docker network and name resolution works, the application container should be able to connect to your database container namedpostgres` within the same network.

Can you post the output of the following two commands, please?

docker ps
docker ps --format "{{.ID}}" | xargs docker inspect | jq '.[].NetworkSettings.Networks'

KnifeHand · March 22, 2023, 5:22am

$ docker ps
CONTAINER ID   IMAGE   COMMAND         CREATED       STATUS     PORTS       NAMES
70b91290ab28   mattermost/mattermost-team-edition:7.1   "/entrypoint.sh matt…"   9 hours ago   Up About a minute (health: starting)   8067/tcp, 0.0.0.0:8065->8065/tcp, :::8065->8065/tcp, 8074-8075/tcp, 0.0.0.0:8443->8443/udp, :::8443->8443/udp   docker_mattermost_1
7dc8981817f9   postgres:13-alpine                       "docker-entrypoint.s…"   9 hours ago   Up 9 hours                             5432/tcp                                                                                      docker_postgres_1

$ docker ps --format "{{.ID}}" | xargs docker inspect | jq '.[].NetworkSettings.Networks'
{
  "docker_default": {
    "IPAMConfig": null,
    "Links": null,
    "Aliases": [
      "70b....",
      "mattermost"
    ],
    "NetworkID": "e1...",
    "EndpointID": "c5...",
    "Gateway": "172.25.0.1",
    "IPAddress": "172.25.0.3",
    "IPPrefixLen": 16,
    "IPv6Gateway": "",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "MacAddress": "02:...",
    "DriverOpts": null
  }
}
{
  "docker_default": {
    "IPAMConfig": null,
    "Links": null,
    "Aliases": [
      "7dc8981817f9",
      "postgres"
    ],
    "NetworkID": "e1d...",
    "EndpointID": "fb5...",
    "Gateway": "172.25.0.1",
    "IPAddress": "172.25.0.2",
    "IPPrefixLen": 16,
    "IPv6Gateway": "",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "MacAddress": "02:....",
    "DriverOpts": null
  }
}

agriesser · March 22, 2023, 7:04am

Alright, this looks good - please use postgres as the name of your database host in the connection string, the connection should work that way and at least not return a connection refused.

KnifeHand · March 22, 2023, 2:33pm

Ok, I set the environment variable in the env.example (line 22) file to postgres.

POSTGRES_USER=mmuser
POSTGRES_PASSWORD=mmuser_password
POSTGRES_DB=postgres

I kept the following variable (line 83) to connect to the DB the same:

## Below one can find necessary settings to spin up the Mattermost container
MM_SQLSETTINGS_DRIVERNAME=postgres
MM_SQLSETTINGS_DATASOURCE=postgres:${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}?sslmode=disable&connect_timeout=10

Unfortunately, I have the same error message result.

{"timestamp":"2023-03-22 14:21:22.985 Z","level":"info","msg":"Server is initializing...","caller":"app/server.go:264","go_version":"go1.18.1"}
{"timestamp":"2023-03-22 14:21:22.985 Z","level":"info","msg":"Pinging SQL","caller":"sqlstore/store.go:228","database":"master"}

agriesser · March 22, 2023, 3:38pm

I cannot see an error message in your logs - Pinging SQL is OK, it’s just a status message here, it does not say that the ping is unsuccessful. Do you have any other logs after that?

KnifeHand · March 22, 2023, 3:50pm

When I run docker logs mattermost on the container this is the only log that repeats.

agriesser · March 22, 2023, 4:03pm

OK, can you please try to start the container in foreground? If you use docker-compose, then please run:

docker-compose down
docker-compose -f docker-compose.yml -f docker-compose.without-nginx.yml up

(no -d at the end of the up and you should stay in foreground and hopefully see more logs).
There’s also a logfile called mattermost.log which should be available in the subdirectory volumes/app/mattermost/logs/ (relative to your docker-compose files), maybe this file contains more information.

KnifeHand · March 22, 2023, 5:29pm

Here is the output for compose running in the foreground:

The same error message exists in the mattermost.log file where no new information besides that error exists.

agriesser · March 22, 2023, 5:45pm

It is again trying to connect to 127.0.0.1:5432, not postgres:5432.
I think your .env file is broken, here’s my MM_SQLSETTINGS_DATASOURCE variable, it looks different than yours:

MM_SQLSETTINGS_DATASOURCE=postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}?sslmode=disable&connect_timeout=10

Can you please use this line in your .env file and try again, then?

KnifeHand · March 22, 2023, 6:37pm

Each time I make an edit to the env.example file, I copy it to the .env.

I had to take out the double slashes ‘//’ before ${POSTGRES_USER} due to a parsing error. However, when I keep the slashes and run docker compose up on my local machine, no problems.

KnifeHand · March 22, 2023, 9:50pm

Using the given environment variable:

I get the following result with the parse error:

root@ip-10-200-1-5:~/mattermost-team-edition/docker# docker-compose -f docker-compose.yml -f docker-compose.without-nginx.yml up
Starting docker_postgres_1 ... done
Recreating docker_mattermost_1 ... done
Attaching to docker_postgres_1, docker_mattermost_1
postgres_1    |
postgres_1    | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres_1    |
postgres_1    | 2023-03-22 21:49:46.242 UTC [1] LOG:  starting PostgreSQL 13.10 on x86_64-pc-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r4) 12.2.1 20220924, 64-bit
postgres_1    | 2023-03-22 21:49:46.242 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres_1    | 2023-03-22 21:49:46.243 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres_1    | 2023-03-22 21:49:46.245 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1    | 2023-03-22 21:49:46.251 UTC [23] LOG:  database system was shut down at 2023-03-22 19:54:04 UTC
postgres_1    | 2023-03-22 21:49:46.256 UTC [1] LOG:  database system is ready to accept connections
mattermost_1  | {"timestamp":"2023-03-22 21:49:46.928 Z","level":"info","msg":"Server is initializing...","caller":"platform/service.go:165","go_version":"go1.19.5"}
mattermost_1  | {"timestamp":"2023-03-22 21:49:46.928 Z","level":"info","msg":"Pinging SQL","caller":"sqlstore/store.go:240","database":"master"}
mattermost_1  | {"timestamp":"2023-03-22 21:49:46.928 Z","level":"error","msg":"Failed to ping DB","caller":"sqlstore/store.go:250","error":"parse \"postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}?sslmode=disable&connect_timeout=10\": net/url: invalid userinfo","retrying in seconds":10}
^CGracefully stopping... (press Ctrl+C again to force)
Stopping docker_mattermost_1   ... done
Stopping docker_postgres_1     ... done

agriesser · March 23, 2023, 4:03am

OK, now we get closer to the real problem. I guess that your docker-compose version is too old and is unable to use variable substitution and therefore fails. The first working version is 1.26.0, yours is probably older.
If you’re using docker-compose from your distribution and the package is outdated and you cannot upgrade it, then please remove it and download the binary. If you have the binary distribution already, replace it with a newer version:

wget https://github.com/docker/compose/releases/download/v2.17.0/docker-compose-linux-x86_64 -O /usr/local/bin/docker-compose
chmod 755 /usr/local/bin/docker-compose

Topic		Replies	Views
Posgres New Install on Ubuntu 20.04 - Failed to Ping DB Troubleshooting	8	2191	April 19, 2021
Installation problems: Failed to ping DB Troubleshooting	10	5554	April 13, 2022
Failed to ping DB error, while installing Troubleshooting	1	645	March 9, 2020
Unable to ping database during installation - Solved Troubleshooting	6	2359	December 4, 2019
Postgress sslmode fails to ping db Troubleshooting	0	818	October 13, 2019

"Failed to ping DB" when trying to start mattermost server

Summary

Expected behavior

Observed behavior

Related Topics