Setting up using Docker

Troubleshooting
41

There should only be one .env file. It contains environment variables for the Mattermost application container, the nginx container (optional) and the PostgreSQL database container, so with the full-blown use, you will have three containers running, which is what we’re going to do now.

The scripts/issue-certificate.sh script should have created the certificates below the current working directory in a folder called “certs”, so let’s assume you’re docker base directory is /home/jpzone/mattermost, the .env file should be at the same level as the certs folder then.
You should then see a subfolder with the relevant *.pem files in certs/etc/letsencrypt/live/YOUR_MM_DOMAIN/ and the .env file already contains the configuration for that, you just need to make sure the correct lines are uncommented.

By default, it looks like this:

CERT_PATH=./volumes/web/cert/cert.pem
KEY_PATH=./volumes/web/cert/key-no-password.pem
#GITLAB_PKI_CHAIN_PATH=<path_to_your_gitlab_pki>/pki_chain.pem
#CERT_PATH=./certs/etc/letsencrypt/live/${DOMAIN}/fullchain.pem
#KEY_PATH=./certs/etc/letsencrypt/live/${DOMAIN}/privkey.pem

CERT_PATH and KEY_PATH need to be available from where you start the docker compose command, so in this example, you would have to change the lines in the .env file so that the second occurences are active:

#CERT_PATH=./volumes/web/cert/cert.pem
#KEY_PATH=./volumes/web/cert/key-no-password.pem
#GITLAB_PKI_CHAIN_PATH=<path_to_your_gitlab_pki>/pki_chain.pem
CERT_PATH=./certs/etc/letsencrypt/live/${DOMAIN}/fullchain.pem
KEY_PATH=./certs/etc/letsencrypt/live/${DOMAIN}/privkey.pem
42

Think I see what’s wrong. The bash scripts/issue-certificate.sh -d <YOUR_MM_DOMAIN> -o ${PWD}/certs script did not create anything. Tried it again and it says no such file or directory. Should I create it manually if that is the case?

/home/jpzone282/mattermost/certs/
/home/jpzone282/mattermost/certs/scripts
/home/jpzone282/mattermost/certs/scripts/issue-certificate.sh

Also just checking my .env is in the following path.

/home/jpzone282/mattermost
43

The directory structure is not correct, the issue-certificate.sh script should not be inside the certs folder, that’s why you cannot execute it right now.
If this is what you’re currently seeing, please run the following commands:

cd /home/jpzone282/mattermost
mv certs/scripts .
scripts/issue-certificate.sh -d <YOUR_MM_DOMAIN> -o ${PWD}/certs

And please do not forget to replace <YOUR_MM_DOMAIN> with the domain name you want to create the certificate for. Since this process will start a let’s encrypt certificate generation, the domain you specified needs to point to the server already and port 80 needs to be open from the internet for the validation to complete.
If you encounter any errors during this process, please paste the exact command and the full error messages in your next message.

44

Where should this be run from? bash scripts/issue-certificate.sh -d mattermost.mywebsite.com -o ${PWD}/certs?

I keep on getting the following error
bash: scripts/issue-certificate.sh: No such file or directory

45

This needs to be run in the directory /home/jpzone282/mattermost, as you can see in my previous post, but that only works if you also moved the scripts directory with the mv command.
Let’s try this again:

cd /home/jpzone282/mattermost
test -d certs/scripts && mv certs/scripts .
bash scripts/issue-certificate.sh -d <YOUR_MM_DOMAIN> -o ${PWD}/certs

Please post the full terminal log (commans you typed + repsonses) if this still doesn’t work then.

46

If I run the following this is what comes up.

bash scripts/issue-certificate.sh -d mattermost.mywebsite.com -o ${PWD}/certs

‘-bash: mattermost.mywebsite.com: No such file or directory’

47

I absolutely cannot reproduce that on my end and am pretty sure that there is a problem with your inputs.
It is very important that you send us the full console log so we can see what’s wrong here.
F.ex. I’m connected to my demo system with a SSH client now (I’m using PuTTY on Windows) and this is what my console logs look like when I try to run the command:

root@myserver:~/mattermost/mm-6.7.0# bash scripts/issue-certificate.sh -d mattermost.mywebsite.com -o ${PWD}/certs
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for mattermost.mywebsite.com

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/mattermost.mywebsite.com-0001/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/mattermost.mywebsite.com-0001/privkey.pem
This certificate expires on 2022-11-26.
These files will be updated when the certificate renews.

NEXT STEPS:
- The certificate will need to be renewed before it expires. Certbot can automatically renew the certificate in the background, but you may need to take steps to enable that functionality. See https://certbot.org/renewal-setup for instructions.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
root@myserver:~/mattermost/mm-6.7.0#

Please post the same output, the shell, the user, the command you typed, including the full surrounding error messages in your next reply.
Also, please additionally run the commands:

ls -1
find certs
find scripts

and post the outputs of them.

48

These are outputs when I run the following commands

user@mysystem:~/mattermost$ bash scripts/issue-certificate.sh -d mattermost.mysebsite.com -o ${PWD}/certs
bash: scripts/issue-certificate.sh: No such file or directory

user@mysystem:~/mattermost$ ls -1
contrib
docker-compose.nginx.yml
docker-compose.without-nginx.yml
docker-compose.yml
docs
env.example
LICENSE
nginx
README.md
scripts
volumes

user@mysystem:~/mattermost$ find certs
find: ‘certs’: No such file or directory

user@mysystem:~/mattermost$ find scripts
scripts
scripts/issue-certificate.sh
scripts/UPGRADE.md
scripts/upgrade-postgres.sh
49

Hmm… please try the following commands:

cd /home/jpzone282/mattermost
file scripts/issue-certificate.sh
./scripts/issue-certificate.sh -d mattermost.mysebsite.com -o ${PWD}/certs
50

Got this but still no certs directory

docker: Error response from daemon: driver failed programming external connectivity on endpoint certbot (95e3716a7981620b724777dcede1ba1b436f8446b583055f31f7597afd06450d): Bind for 0.0.0.0:80 failed: port is already allocated.
ERRO[0000] error waiting for container: context canceled
sed: can't read /home/myuser/mattermost/certs/etc/letsencrypt/renewal/mattermost.mywebsite.com.conf: No such file or directory
tee: /home/myuser/mattermost/certs/etc/letsencrypt/renewal/mattermost.mywebsite.com.conf: No such file or directory
51

OK, we’re making progress now.
The certificate can only be issued, when there’s nothing else running on port 80 of your server currently.
The error message in the first line says, that port 80 is blocked by another application, maybe by another nginx container.
You can find out what process is using the port with the following commands:

lsof -n -i :80
docker ps

Please send the output of these two commands.

52

Ah yeah i do have another container that is running on port 80

53

The Let’s Encrypt Container used for the issue-certificate.sh script will need to have access to this port for a few seconds, so if possible, stop the other container, run the process again and this will give you the new certificate then.
Once done, you can restart the other container on port 80, but this will only postpone your problem, because the nginx container in the Mattermost distribution also wants to listen to port 80 (but you said you already changed that, so you should be good).

54

Okay worked i think. But think I am still having a conflict somewhere.
This is my mattermost nginx ports

0.0.0.0:1443  443/tcp
0.0.0.0:1180  80/tcp

NGINX Mattermost log

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] duplicate upstream "backend" in /etc/nginx/conf.d/nginx.conf:1
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] duplicate upstream "backend" in /etc/nginx/conf.d/nginx.conf:1
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] duplicate upstream "backend" in /etc/nginx/conf.d/nginx.conf:1
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] duplicate upstream "backend" in /etc/nginx/conf.d/nginx.conf:1
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] duplicate upstream "backend" in /etc/nginx/conf.d/nginx.conf:1
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] duplicate upstream "backend" in /etc/nginx/conf.d/nginx.conf:1
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] duplicate upstream "backend" in /etc/nginx/conf.d/nginx.conf:1
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] duplicate upstream "backend" in /etc/nginx/conf.d/nginx.conf:1
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] duplicate upstream "backend" in /etc/nginx/conf.d/nginx.conf:1
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] duplicate upstream "backend" in /etc/nginx/conf.d/nginx.conf:1
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] duplicate upstream "backend" in /etc/nginx/conf.d/nginx.conf:1
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] duplicate upstream "backend" in /etc/nginx/conf.d/nginx.conf:1
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] duplicate upstream "backend" in /etc/nginx/conf.d/nginx.conf:1
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
nginx: [emerg] duplicate upstream "backend" in /etc/nginx/conf.d/nginx.conf:1
55

In this comment you said that you manually added the configuration to an nginx.conf file. The error message here now looks like this change is still active, although it shouldn’t be.
Did you modify the docker-compose.nginx.yml file or the files below the nginx subdirectory in the docker directory?

Please run the following commands:

cd /home/jpzone282/mattermost
cat docker-compose.nginx.yml
find nginx

And please post the outputs again.

56

I haven’t modified the docker-compose.nginx.yml


version: "2.4"

services:
  nginx:
    depends_on:
      - mattermost
    container_name: nginx_mattermost
    image: nginx:${NGINX_IMAGE_TAG}
    restart: ${RESTART_POLICY}
    security_opt:
      - no-new-privileges:true
    pids_limit: 100
    read_only: true
    tmpfs:
      - /var/run
      - /var/cache
      - /var/log/nginx
    volumes:
      - ${NGINX_CONFIG_PATH}:/etc/nginx/conf.d:ro
      - ${NGINX_DHPARAMS_FILE}:/dhparams4096.pem
      - ${CERT_PATH}:/cert.pem:ro
      - ${KEY_PATH}:/key.pem:ro
      - shared-webroot:/usr/share/nginx/html
      - ./nginx/conf.d/nginx.conf:/etc/nginx/site-available/mattermost
    environment:
      # timezone inside container
      - TZ
    ports:
      - ${HTTPS_PORT}:443
      - ${HTTP_PORT}:80

# Shared volume for Let's Encrypt certificate renewal with a webroot
volumes:
  shared-webroot:
    name: shared-webroot

# This network name is being used for Let's Encrypt certificate renewal
networks:
  default:
    name: mattermost



nginx/
nginx/conf.d
nginx/conf.d/default.conf
nginx/conf.d/nginx.conf
nginx/dhparams4096.pem
57

There should not be a file called “nginx.conf” in this subdirectory.
Please run:

cd /home/jpzone282/mattermost
mv nginx/conf.d/nginx.conf nginx/conf.d/nginx.conf.disabled

and try again then.

58

it cant find the certificates now

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration


/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/


/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh


10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)


/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh


/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh


/docker-entrypoint.sh: Configuration complete; ready for start up


nginx: [emerg] cannot load certificate "/cert.pem": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)


/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration


/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/


/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh


10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)


/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh


/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh


/docker-entrypoint.sh: Configuration complete; ready for start up


nginx: [emerg] cannot load certificate "/cert.pem": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)


/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration


/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/


/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh


10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)


/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh


/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh


/docker-entrypoint.sh: Configuration complete; ready for start up


nginx: [emerg] host not found in upstream "mattermost:8065" in /etc/nginx/conf.d/default.conf:12
59

Alright, getting closer, please run:

cd /home/jpzone282/mattermost
CERT=$(awk -F= '$1~/^CERT_PATH/ { print $2 }' .env); find $CERT; file $CERT; ls -l $CERT; head -3 $CERT

It looks as if previous runs have created empty directories where the files should be, but to be sure, I need to see the outputs.
Also the nginx container requires a working mattermost application container, otherwise the name cannot be resolved (last line in your output).

60 
user@mysystem:~/mattermost$ CERT=$(awk -F= '$1~/^CERT_PATH/ { print $2 }' .env); find $CERT; file $CERT; ls -l $CERT; head -3 $CERT
./volumes/web/cert/cert.pem
find: ‘./certs/etc/letsencrypt/live/${DOMAIN}/fullchain.pem’: Permission denied
./volumes/web/cert/cert.pem:                          directory
./certs/etc/letsencrypt/live/${DOMAIN}/fullchain.pem: cannot open `./certs/etc/letsencrypt/live/${DOMAIN}/fullchain.pem' (Permission denied)
ls: cannot access './certs/etc/letsencrypt/live/${DOMAIN}/fullchain.pem': Permission denied
./volumes/web/cert/cert.pem:
total 0
==> ./volumes/web/cert/cert.pem <==
head: error reading './volumes/web/cert/cert.pem': Is a directory
head: cannot open './certs/etc/letsencrypt/live/${DOMAIN}/fullchain.pem' for reading: Permission denied