MM docker install traefik/Cloudflare Gateway time-out

Troubleshooting
1

Summary
New MM installation using docker behind traefik connection timeout when trying to reach mattermost.example.com

Steps to reproduce

  1. Mostly copied the mattermost docker compose example
  2. added parts to make it work with traefik as a reverese proxy. ( mostly labels )
  3. run the container with docker compose up -d

Expected behavior
mattermost is accessable via mattermost.example.com (redacted)

Observed behavior
getting messsage from cloudflare ( hoster of the domain and used for DNS challenge)

cloudflare_gateway_timeout
cloudflare_gateway_timeout922×503 38.5 KB

the logs show:mattermost logs

Details
I got a valid certificate from traefik/cloudflare, therefore I assume the problem is not related to the certificate, but rather docker network or port related.

traefik_mattermost_docker
traefik_mattermost_docker1427×765 47.3 KB

the full compose.yaml:

services:
  postgres:
    image: postgres:${POSTGRES_IMAGE_TAG}
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    pids_limit: 100
    read_only: true
    tmpfs:
      - /tmp
      - /var/run/postgresql
    volumes:
      - ${POSTGRES_DATA_PATH}:/var/lib/postgresql/data
    environment:
      # timezone inside container
      - TZ
      - POSTGRES_USER
      - POSTGRES_PASSWORD
      - POSTGRES_DB
    networks:
      - default
      - proxy
  mattermost:
    depends_on:
      - postgres
    image: mattermost/${MATTERMOST_IMAGE}:${MATTERMOST_IMAGE_TAG}
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    pids_limit: 200
    read_only: ${MATTERMOST_CONTAINER_READONLY}
    tmpfs:
      - /tmp
    volumes:
      - ${MATTERMOST_CONFIG_PATH}:/mattermost/config:rw
      - ${MATTERMOST_DATA_PATH}:/mattermost/data:rw
      - ${MATTERMOST_LOGS_PATH}:/mattermost/logs:rw
      - ${MATTERMOST_PLUGINS_PATH}:/mattermost/plugins:rw
      - ${MATTERMOST_CLIENT_PLUGINS_PATH}:/mattermost/client/plugins:rw
      - ${MATTERMOST_BLEVE_INDEXES_PATH}:/mattermost/bleve-indexes:rw
      # When you want to use SSO with GitLab, you have to add the cert pki chain of GitLab inside Alpine
      # to avoid Token request failed: certificate signed by unknown authority 
      # (link: https://github.com/mattermost/mattermost-server/issues/13059 and https://github.com/mattermost/docker/issues/34)
      # - ${GITLAB_PKI_CHAIN_PATH}:/etc/ssl/certs/pki_chain.pem:ro
    environment:
      # timezone inside container
      - TZ
      - MM_SQLSETTINGS_DRIVERNAME
      - MM_SQLSETTINGS_DATASOURCE
      - MM_BLEVESETTINGS_INDEXDIR
      - MM_SERVICESETTINGS_SITEURL
    labels:
      - traefik.enable=true
      - traefik.http.routers.mattermost.rule=Host(`mattermost.example.org`)
      - traefik.http.routers.mattermost.entryPoints=https
      - traefik.http.routers.mattermost.tls=true
      - traefik.http.routers.mattermost.service=mattermost
      - traefik.http.routers.mattermost.tls.certresolver=cloudflare
      - traefik.http.services.mattermost.loadbalancer.server.port=8065
networks:
  proxy:
    external: true

.env

DOMAIN=mattermost.example.org

TZ=Europe/Vienna

#RESTART_POLICY=unless-stopped
POSTGRES_IMAGE_TAG=13-alpine
POSTGRES_DATA_PATH=./volumes/db/var/lib/postgresql/data

POSTGRES_USER=mmuser
POSTGRES_PASSWORD=mmuser_password
POSTGRES_DB=mattermost

HTTPS_PORT=443
HTTP_PORT=80
CALLS_PORT=8443

MATTERMOST_CONFIG_PATH=./volumes/app/mattermost/config
MATTERMOST_DATA_PATH=./volumes/app/mattermost/data
MATTERMOST_LOGS_PATH=./volumes/app/mattermost/logs
MATTERMOST_PLUGINS_PATH=./volumes/app/mattermost/plugins
MATTERMOST_CLIENT_PLUGINS_PATH=./volumes/app/mattermost/client/plugins
MATTERMOST_BLEVE_INDEXES_PATH=./volumes/app/mattermost/bleve-indexes
MM_BLEVESETTINGS_INDEXDIR=/mattermost/bleve-indexes

MATTERMOST_IMAGE=mattermost-team-edition
MATTERMOST_IMAGE_TAG=8.1.9

MATTERMOST_CONTAINER_READONLY=false

APP_PORT=8065

MM_SQLSETTINGS_DRIVERNAME=postgres
MM_SQLSETTINGS_DATASOURCE=postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}?sslmode=disable&connect_timeout=10

MM_SERVICESETTINGS_SITEURL=https://${DOMAIN}

When starting the container (not a full recreate as shown in the extensive log above),
I see similar logs.

mattermost-postgres-1    | 
mattermost-postgres-1    | PostgreSQL Database directory appears to contain a database; Skipping initialization
mattermost-postgres-1    | 
mattermost-postgres-1    | 2024-06-22 19:32:15.974 CEST [1] LOG:  starting PostgreSQL 13.15 on x86_64-pc-linux-musl, compiled by gcc (Alpine 13.2.1_git20240309) 13.2.1 20240309, 64-bit
mattermost-postgres-1    | 2024-06-22 19:32:15.974 CEST [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
mattermost-postgres-1    | 2024-06-22 19:32:15.974 CEST [1] LOG:  listening on IPv6 address "::", port 5432
mattermost-postgres-1    | 2024-06-22 19:32:15.975 CEST [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
mattermost-postgres-1    | 2024-06-22 19:32:15.980 CEST [28] LOG:  database system was shut down at 2024-06-22 19:32:15 CEST
mattermost-postgres-1    | 2024-06-22 19:32:15.989 CEST [1] LOG:  database system is ready to accept connections
mattermost-mattermost-1  | {"timestamp":"2024-06-22 19:36:16.836 +02:00","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","method":"GET","url":"/api/v4/system/ping","request_id":"gf1tgitdib867ghsjqcynrohfw","status_code":"200"}

Not sure why postgres is listening on 5432 but that port is not mentioned in the .env file.

I saw the last line of the log in a similar issue, but there was no explaination if it is related to the issue
Not sure where to go from here. I’m poking around, but I am out of good Ideas.