Server deployment issues

I have followed the instructions closely to install matter most as a docker on my server. However my browser cannot connect to the matter most server. If i type docker ps after starting the server is appears to be constantly restarting.

CONTAINER ID   IMAGE                                          COMMAND                  CREATED         STATUS                          PORTS      NAMES
812363936ce6   nginx:alpine                                   "/docker-entrypoint.…"   9 minutes ago   Restarting (1) 35 seconds ago              nginx_mattermost
7469c9831860   mattermost/mattermost-enterprise-edition:6.3   "/entrypoint.sh matt…"   9 minutes ago   Restarting (1) 20 seconds ago              docker_mattermost_1
7df55a0e1b18   postgres:13-alpine                             "docker-entrypoint.s…"   9 minutes ago   Up 9 minutes                    5432/tcp   docker_postgres_1

any ideas what’s wrong?

P.S. I have checked that my domain resolves correctly and use the following to launch my server

sudo docker-compose -f docker-compose.yml -f docker-compose.nginx.yml up -d

Hi tobor,

what parts of your .env file did you change? Only the domain? Running the docker compose command without creating the SSL certificates first (you will not see this in the short deploy documentation, this needs to be done manually) usually is a reason why the nginx container is restarting all the time.

You can try to debug that by running the command docker logs <containerid> to see what’s going on. I’ve prepared a setup here with a broken nginx container, this is what docker ps looks like:

# docker ps
CONTAINER ID   IMAGE                                          COMMAND                  CREATED         STATUS                          PORTS                               NAMES
eb94836985da   nginx:alpine                                   "/docker-entrypoint.…"   2 minutes ago   Restarting (1) 46 seconds ago                                       nginx_mattermost
88f4f2c50fcf   mattermost/mattermost-enterprise-edition:6.3   "/entrypoint.sh matt…"   2 minutes ago   Up 2 minutes (healthy)          8065/tcp, 8067/tcp, 8074-8075/tcp   mm-711-mattermost-1
bb0bbb33d085   postgres:13-alpine                             "docker-entrypoint.s…"   2 minutes ago   Up 2 minutes                    5432/tcp                            mm-711-postgres-1

As you can see, container eb94836985da is in “restarting” mode, so you can check the logs using this command:

root@ag-bitch0101:~/mattermost/mm-7.1.1# docker logs eb94836985da
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] cannot load certificate "/cert.pem": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)

And in my case you can see, that it’s having problems with loading the certificate and is therefore restarting.

What’s the output of your docker logs command for both restarting containers?
Also you MIGHT need a newer docker version, because the mattermost docker build works with variable substitutions in the docker-compose files and at least the docker-compose binaries that ship with Debian by default do not support that, so I had to use the versions from download.docker.com (you can find instructions for all majjor operating systems online, here the link for Debian f.ex.: Install Docker Engine on Debian | Docker Documentation ).

Once done, please note that it’s not docker-compose anymore, but docker compose (it’s now a parameter to the docker binary and not a standalone binary anymore).

But that’s just guessing for the reasons without seeing the logs, but out of my experience, these are the two major reasons why containers are restarting on a fresh deployment.

On another sidenote: The docker deployment still defaults to Mattermost 6.3 and the enterprise version, so depending on your requirements, you might also want to change these two lines in the .env file:

## This will be 'mattermost-enterprise-edition' or 'mattermost-team-edition' based on the version of Mattermost you're installing.
MATTERMOST_IMAGE=mattermost-enterprise-edition
MATTERMOST_IMAGE_TAG=6.3

The most recent version (released yesterday) is 7.1.1.

Many thanks for replying to my issue. In the .env file i added my domain name and uncommented the 2 lines which are the paths to my certificates.

error logs for nginx:alpine are

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] cannot load certificate "/cert.pem": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] cannot load certificate "/cert.pem": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] cannot load certificate "/cert.pem": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] cannot load certificate "/cert.pem": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] cannot load certificate "/cert.pem": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] cannot load certificate "/cert.pem": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] cannot load certificate "/cert.pem": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] host not found in upstream "mattermost:8065" in /etc/nginx/conf.d/default.conf:12
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] host not found in upstream "mattermost:8065" in /etc/nginx/conf.d/default.conf:12

error logs for mattermost/mattermost-enterprise-edition:6.3 are

Error: failed to load configuration: failed to create store: unable to load on store creation: invalid config: Config.IsValid: model.config.is_valid.site_url.app_error, 
Usage:
  mattermost [flags]
  mattermost [command]

looks like a certificate issue. The paths are correct.

This indicates that the format of the file is incorrect. Please make sure that your certificate files are in BASE64 format (PEM) and that they are real files and no directories (sometimes this is happening when you start docker first before having the files in place):

$ mm-6.7.0/volumes/web/cert# ls -lh
total 12K
-rw-r--r-- 1 2000 2000 5.5K May 18 07:33 cert.pem
-rw------- 1 2000 2000 1.7K May 18 07:34 key-no-password.pem

$ mm-6.7.0/volumes/web/cert# head -2 cert.pem && echo "[...]" && tail -2 cert.pem
-----BEGIN CERTIFICATE-----
MIIFJzCCBA+gAwIBAgISBGaFi17tXWztClOAzmqFkLPWMA0GCSqGSIb3DQEBCwUA
[...]
Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5
-----END CERTIFICATE-----

mm-6.7.0/volumes/web/cert# head -2 key-no-password.pem && echo "[...]" && tail -2 key-no-password.pem
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCy//cY2OWv1N2h
[...]
m2QuC7ZoV3PPJdNHqmsHQVSJ
-----END PRIVATE KEY-----

The error message says that your file is in a different format, so please make sure it looks similar to my output on your end.
Also for the private key, it needs to read “BEGIN PRIVATE KEY” and not “BEGIN ENCRYPTED PRIVATE KEY” - the nginx configuration here expects a private key without passphrase.

This is the indication for the docker-compose plugin being too old.
You will need to update to the docker compose version I mentioned earlier and ditch your docker-compose setup.

Alternativelity, you can remove all the variable references in your .env file and place the values directly where they are needed.

Currently, it looks like this:

DOMAIN=yourdomain.com
POSTGRES_USER=mmuser
POSTGRES_PASSWORD=mmuser_password
POSTGRES_DB=mattermost
[...]
MM_SQLSETTINGS_DATASOURCE=postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}?sslmode=disable&connect_timeout=10
MM_SERVICESETTINGS_SITEURL=https://${DOMAIN}

And the problem here is, that the variables MM_SQLSETTINGS_DATASOURCE as well as MM_SERVICESETTINGS_SITEURL cannot be filled correctly, because your docker-compose version does not support variables as values for variables, so instead of MM_SERVICESETTINGS_SITEURL being set to https://yourdomain.com, it is being set to https:// and this value is invalid.

So if you cannot or do not want to upgrade your docker setup, you can just fix that in the .env file by making sure that you replace all variables manually, so based on the above example, the two settings MM_SQLSETTINGS_DATASOURCE and MM_SERVICESETTINGS_SITEURL will have to look like this:

MM_SQLSETTINGS_DATASOURCE=postgres://mmuser:mmuser_password@postgres:5432/mattermost?sslmode=disable&connect_timeout=10
MM_SERVICESETTINGS_SITEURL=https://yourdomain.com

Thanks again for your help. Unfortunately that’s all above my skill set at the moment, so i won’t be running MatterMost on my own server for some time. Amazing how many issues there are.

I’m sorry to hear that, especially since you’re not far away from a working deployment.
The only two issues here are a wrong format of your certificate files (obviously, maybe some garbage in them or an unsupported format, so they would need to be converted, but I will need to see the starting lines of the contents of the files first in order to understand what format we have and how they need to be converted) and an outdated docker-compose version, which can be easily workarounded by populating the MM_* variables without the use of the other variables.

So if you change your mind, let me know, we’re here to help.

Hi @tobor ,

I stumbled upon another way to install mattermost today, the Omnibus Installation. If you are currently running Ubuntu 18.04 or Ubuntu 20.04, following the instructions in the previous link should get you going almost instantly; the Omnibus installation includes all the relevant configuration steps for you and brought me to a running native installation within 3 minutes. Worth a try…

Hi @agriesser

Thank you so much.
After weeks of banging my head on a wall, that method worked in 3 minutes. All up and running now

Cheers

Awesome :slight_smile: Happy Mattermosting then!