Mattermost docker: connect() failed (111: Connection refused) while connecting to upstream

Troubleshooting
1

I have no idea why, my Mattermost instance was working fine just fine, and then stopped working with error messages like those:

[error] 18#18: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: , request: "GET /api/v4/websocket?connection_id=mkp3yo7sitdxuny1a9ec4ei8ao&sequence_number=294 HTTP/1.1", upstream: "http://172.19.0.4:8000/api/v4/websocket?connection_id=mkp3yo7sitdxuny1a9ec4ei8ao&sequence_number=294", host: "mattermost.dsimb.inserm.fr"

I was manipulating some iptables rules concerning another program a bit but I have no idea what I messed up about mattermost…

Here are the logs of the app, web and db when I launch the docker-compose:

APP

$ docker-compose logs -f app
Attaching to mattermost_app_1
app_1  | Using existing config file /mattermost/config/config.json
app_1  | Using existing database connection
app_1  | Starting mattermost
app_1  | info  [2023-02-03 10:38:26.622 +01:00] Server is initializing...                     caller="platform/service.go:165" go_version=go1.18.1
app_1  | info  [2023-02-03 10:38:26.622 +01:00] EXPERIMENTAL: Starting Bleve                  caller="bleveengine/bleve.go:185"
app_1  | info  [2023-02-03 10:38:26.630 +01:00] Pinging SQL                                   caller="sqlstore/store.go:238" database=master
app_1  | info  [2023-02-03 10:38:26.721 +01:00] Starting websocket hubs                       caller="platform/web_hub.go:95" number_of_hubs=16
app_1  | info  [2023-02-03 10:38:26.725 +01:00] Loaded system translations                    caller="i18n/i18n.go:93" for locale=fr from locale=/mattermost/i18n/fr.json
app_1  | info  [2023-02-03 10:38:26.738 +01:00] Current version is 7.7.1 (7.7.1/Thu Jan 19 00:47:45 UTC 2023/63149c4869898f8c1d3a94ea728370ec4a07575f/none) caller="app/server.go:395" current_version=7.7.1 build_number=7.7.1 build_date="Thu Jan 19 00:47:45 UTC 2023" build_hash=63149c4869898f8c1d3a94ea728370ec4a07575f build_hash_enterprise=none
app_1  | info  [2023-02-03 10:38:26.738 +01:00] Team Edition Build                            caller="app/server.go:406" enterprise_build=false
app_1  | info  [2023-02-03 10:38:26.738 +01:00] Printing current working                      caller="app/server.go:410" directory=/mattermost
app_1  | info  [2023-02-03 10:38:26.738 +01:00] Loaded config                                 caller="app/server.go:411" source="file:///mattermost/config/config.json"
app_1  | info  [2023-02-03 10:38:26.745 +01:00] Starting workers                              caller="jobs/workers.go:48"
app_1  | info  [2023-02-03 10:38:26.745 +01:00] Starting schedulers.                          caller="jobs/schedulers.go:47"
app_1  | error [2023-02-03 10:38:26.749 +01:00] License key from https://mattermost.com required to unlock enterprise features. caller="platform/license.go:106" error="resource: License id: "
app_1  | info  [2023-02-03 10:38:26.895 +01:00] Starting up plugins                           caller="app/plugin.go:213"
app_1  | info  [2023-02-03 10:38:26.895 +01:00] Syncing plugins from the file store           caller="app/plugin.go:294"
app_1  | info  [2023-02-03 10:38:26.932 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/memes.tar.gz
app_1  | info  [2023-02-03 10:38:26.932 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.github.scottleedavis.mattermost-plugin-remind.tar.gz
app_1  | info  [2023-02-03 10:38:26.932 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/focalboard.tar.gz
app_1  | info  [2023-02-03 10:38:26.933 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/playbooks.tar.gz
app_1  | info  [2023-02-03 10:38:26.933 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.mattermost.draw-plugin.tar.gz
app_1  | info  [2023-02-03 10:38:26.933 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.mattermost.calls.tar.gz
app_1  | info  [2023-02-03 10:38:26.933 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.matthewdorner.mattermost-plugin-chess.tar.gz
app_1  | info  [2023-02-03 10:38:26.933 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.mattermost.welcomebot.tar.gz
app_1  | info  [2023-02-03 10:38:26.934 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/mattermost-autolink.tar.gz
app_1  | info  [2023-02-03 10:38:26.934 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.mattermost.copy-post-plugin.tar.gz
app_1  | info  [2023-02-03 10:38:26.934 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/channel-notes.tar.gz
app_1  | info  [2023-02-03 10:38:26.935 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.github.moussetc.mattermost.plugin.diceroller.tar.gz
app_1  | info  [2023-02-03 10:38:26.945 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.github.matterpoll.matterpoll.tar.gz
app_1  | info  [2023-02-03 10:38:26.945 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.github.moussetc.mattermost.plugin.giphy.tar.gz
app_1  | info  [2023-02-03 10:38:26.945 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.mattermost.custom-attributes.tar.gz
app_1  | info  [2023-02-03 10:38:26.955 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.mattermost.agenda.tar.gz
app_1  | info  [2023-02-03 10:38:26.956 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.mattermost.plugin-todo.tar.gz
app_1  | info  [2023-02-03 10:38:26.957 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/github.tar.gz
app_1  | warn  [2023-02-03 10:38:27.872 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.mattermost.welcomebot
app_1  | warn  [2023-02-03 10:38:28.257 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=memes
app_1  | warn  [2023-02-03 10:38:28.258 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.mattermost.custom-attributes
app_1  | warn  [2023-02-03 10:38:28.828 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=mattermost-autolink
app_1  | warn  [2023-02-03 10:38:28.908 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.github.moussetc.mattermost.plugin.giphy
app_1  | error [2023-02-03 10:38:29.949 +01:00] Failed to sync plugin from file store         caller="app/plugin.go:366" bundle=plugins/channel-notes.tar.gz error="installExtractedPlugin: Unable to move plugin from temporary directory to final destination. Another plugin may be using the same directory name., destination already exists"
app_1  | warn  [2023-02-03 10:38:30.437 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.github.scottleedavis.mattermost-plugin-remind
app_1  | info  [2023-02-03 10:38:30.547 +01:00] Post.Message has size restrictions            caller="sqlstore/post_store.go:2483" max_characters=16383 max_bytes=65535
app_1  | warn  [2023-02-03 10:38:30.614 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.mattermost.agenda
app_1  | error [2023-02-03 10:38:30.723 +01:00] Failed to sync plugin from file store         caller="app/plugin.go:366" bundle=plugins/com.github.matterpoll.matterpoll.tar.gz error="installExtractedPlugin: Unable to move plugin from temporary directory to final destination. Another plugin may be using the same directory name., destination already exists"
app_1  | warn  [2023-02-03 10:38:30.839 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.mattermost.calls
app_1  | info  [2023-02-03 10:38:31.071 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:84"
app_1  | info  [2023-02-03 10:38:31.071 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:84"
app_1  | info  [2023-02-03 10:38:31.072 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:84"
app_1  | info  [2023-02-03 10:38:31.072 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:84"
app_1  | info  [2023-02-03 10:38:31.073 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:84"
app_1  | info  [2023-02-03 10:38:31.073 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:84"
app_1  | info  [2023-02-03 10:38:31.074 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:84"
app_1  | info  [2023-02-03 10:38:31.074 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:84"
app_1  | warn  [2023-02-03 10:38:31.324 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=playbooks
app_1  | warn  [2023-02-03 10:38:32.143 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=focalboard
app_1  | info  [2023-02-03 10:38:32.207 +01:00] connectDatabase                               caller="app/plugin_api.go:973" plugin_id=focalboard dbType=postgres
app_1  | error [2023-02-03 10:38:32.222 +01:00] Table creation / migration failed             caller="app/plugin_api.go:976" plugin_id=focalboard error=""pq: column "dirty" does not exist in line 0: SELECT version, dirty FROM "public"."focalboard_schema_migrations" LIMIT 1""
app_1  | info  [2023-02-03 10:38:32.225 +01:00] plugin process exited                         caller="plugin/hclog_adapter.go:61" plugin_id=focalboard wrapped_extras=path/mattermost/plugins/focalboard/server/dist/plugin-linux-amd64pid126
app_1  | error [2023-02-03 10:38:32.248 +01:00] Failed to sync plugin from file store         caller="app/plugin.go:366" bundle=plugins/focalboard.tar.gz error="installExtractedPlugin: Unable to restart plugin on upgrade., error initializing the DB: pq: column "dirty" does not exist in line 0: SELECT version, dirty FROM "public"."focalboard_schema_migrations" LIMIT 1"
app_1  | warn  [2023-02-03 10:38:33.706 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.mattermost.plugin-channel-export
app_1  | info  [2023-02-03 10:38:33.926 +01:00] plugin process exited                         caller="plugin/hclog_adapter.go:61" plugin_id=playbooks wrapped_extras=path/mattermost/plugins/playbooks/server/dist/plugin-linux-amd64pid113
app_1  | info  [2023-02-03 10:38:34.293 +01:00] rtc: server was shutdown                      caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:84"
app_1  | warn  [2023-02-03 10:38:34.703 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.mattermost.apps
app_1  | info  [2023-02-03 10:38:34.873 +01:00] plugin process exited                         caller="plugin/hclog_adapter.go:61" plugin_id=com.mattermost.calls wrapped_extras=path/mattermost/plugins/com.mattermost.calls/server/dist/plugin-linux-amd64pid102
app_1  | info  [2023-02-03 10:38:34.949 +01:00] activated                                     caller="app/plugin_api.go:973" plugin_id=com.mattermost.apps
app_1  | warn  [2023-02-03 10:38:35.491 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=playbooks
app_1  | info  [2023-02-03 10:38:35.748 +01:00] registered collection and topic type          caller="app/collection.go:33" plugin_id=playbooks collection_type=run topic_type=status
app_1  | info  [2023-02-03 10:38:35.748 +01:00] registered collection and topic type          caller="app/collection.go:33" plugin_id=playbooks collection_type=run topic_type=task
app_1  | warn  [2023-02-03 10:38:36.655 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.mattermost.calls
app_1  | warn  [2023-02-03 10:38:36.707 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=focalboard
app_1  | info  [2023-02-03 10:38:36.768 +01:00] connectDatabase                               caller="app/plugin_api.go:973" plugin_id=focalboard dbType=postgres
app_1  | info  [2023-02-03 10:38:36.795 +01:00] {"level":"info","msg":"Pinging SQL","fields":{"database":"master"}}
app_1  |  caller="io/io.go:428" plugin_id=focalboard source=plugin_stderr
app_1  | info  [2023-02-03 10:38:36.836 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:90"
app_1  | info  [2023-02-03 10:38:36.837 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:90"
app_1  | info  [2023-02-03 10:38:36.837 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:90"
app_1  | info  [2023-02-03 10:38:36.838 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:90"
app_1  | info  [2023-02-03 10:38:36.838 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:90"
app_1  | info  [2023-02-03 10:38:36.839 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:90"
app_1  | info  [2023-02-03 10:38:36.839 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:90"
app_1  | info  [2023-02-03 10:38:36.840 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:90"
app_1  | info  [2023-02-03 10:38:36.871 +01:00] Initialized notification backend              caller="app/plugin_api.go:973" plugin_id=focalboard name=notifyMentions
app_1  | info  [2023-02-03 10:38:36.872 +01:00] Initialized notification backend              caller="app/plugin_api.go:973" plugin_id=focalboard name=notifySubscriptions
app_1  | info  [2023-02-03 10:38:36.872 +01:00] Initialized notification backend              caller="app/plugin_api.go:973" plugin_id=focalboard name=notifyLogger
app_1  | info  [2023-02-03 10:38:36.888 +01:00] FocalBoard Server                             caller="app/plugin_api.go:973" plugin_id=focalboard version=7.7.0 edition=plugin build_number=3856523067 build_date=""Fri Jan  6 15:51:58 UTC 2023"" build_hash=938e7f1c82c7af00115248e284f302c28557f040
app_1  | info  [2023-02-03 10:38:36.888 +01:00] Server.Start                                  caller="app/plugin_api.go:973" plugin_id=focalboard
app_1  | info  [2023-02-03 10:38:36.908 +01:00] Boards product successfully started.          caller="app/plugin_api.go:973" plugin_id=focalboard
app_1  | info  [2023-02-03 10:38:36.958 +01:00] Starting Server...                            caller="app/server.go:878"
app_1  | info  [2023-02-03 10:38:36.958 +01:00] RateLimiter is enabled                        caller="app/server.go:912"
app_1  | info  [2023-02-03 10:38:36.959 +01:00] Server is listening on [::]:8065              caller="app/server.go:950" address="[::]:8065"

DB

$ docker-compose logs -f db
Attaching to mattermost_db_1
db_1   | AWS_ACCESS_KEY_ID is required for Wal-E but not set. Skipping Wal-E setup.
db_1   | AWS_SECRET_ACCESS_KEY is required for Wal-E but not set. Skipping Wal-E setup.
db_1   | WALE_S3_PREFIX is required for Wal-E but not set. Skipping Wal-E setup.
db_1   | AWS_REGION is required for Wal-E but not set. Skipping Wal-E setup.
db_1   |
db_1   | PostgreSQL Database directory appears to contain a database; Skipping initialization
db_1   |
db_1   | 2023-02-03 09:38:25.059 UTC [14] LOG:  starting PostgreSQL 13.2 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.2.1_pre1) 10.2.1 20201203, 64-bit
db_1   | 2023-02-03 09:38:25.059 UTC [14] LOG:  listening on IPv4 address "0.0.0.0", port 5432
db_1   | 2023-02-03 09:38:25.059 UTC [14] LOG:  listening on IPv6 address "::", port 5432
db_1   | 2023-02-03 09:38:25.062 UTC [14] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1   | 2023-02-03 09:38:25.096 UTC [29] LOG:  database system was interrupted; last known up at 2023-02-03 09:34:47 UTC
db_1   | 2023-02-03 09:38:25.137 UTC [29] LOG:  database system was not properly shut down; automatic recovery in progress
db_1   | 2023-02-03 09:38:25.139 UTC [29] LOG:  redo starts at 8/AC83E28
db_1   | 2023-02-03 09:38:25.139 UTC [29] LOG:  invalid record length at 8/AC8C0A8: wanted 24, got 0
db_1   | 2023-02-03 09:38:25.139 UTC [29] LOG:  redo done at 8/AC8C080
db_1   | 2023-02-03 09:38:25.152 UTC [14] LOG:  database system is ready to accept connections
db_1   | 2023-02-03 09:38:30.548 UTC [44] ERROR:  duplicate key value violates unique constraint "users_email_key"
db_1   | 2023-02-03 09:38:30.548 UTC [44] DETAIL:  Key (email)=(remindbot@localhost) already exists.
db_1   | 2023-02-03 09:38:30.548 UTC [44] STATEMENT:  INSERT INTO Users
db_1   | 			(Id, CreateAt, UpdateAt, DeleteAt, Username, Password, AuthData, AuthService,
db_1   | 				Email, EmailVerified, Nickname, FirstName, LastName, Position, Roles, AllowMarketing,
db_1   | 				Props, NotifyProps, LastPasswordUpdate, LastPictureUpdate, FailedAttempts,
db_1   | 				Locale, Timezone, MfaActive, MfaSecret, RemoteId)
db_1   | 			VALUES
db_1   | 			($1, $2, $3, $4, $5, $6, $7, $8,
db_1   | 				$9, $10, $11, $12, $13, $14, $15, $16,
db_1   | 				$17, $18, $19, $20, $21,
db_1   | 				$22, $23, $24, $25, $26)
db_1   | 2023-02-03 09:38:32.221 UTC [46] ERROR:  column "dirty" does not exist at character 17
db_1   | 2023-02-03 09:38:32.221 UTC [46] STATEMENT:  SELECT version, dirty FROM "public"."focalboard_schema_migrations" LIMIT 1

WEB (truncated)

$ docker-compose logs -f web
Attaching to mattermost_web_1
web_1  | linking plain config
web_1  | 2023/02/03 10:38:25 [error] 18#18: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: , request: "GET /api/v4/websocket?connection_id=mkp3yo7sitdxuny1a9ec4ei8ao&sequence_number=294 HTTP/1.1", upstream: "http://172.19.0.4:8000/api/v4/websocket?connection_id=mkp3yo7sitdxuny1a9ec4ei8ao&sequence_number=294", host: "mattermost.dsimb.inserm.fr"

My docker-compose.yaml file:

version: "3"

services:

  db:
    build: db
    read_only: true
    restart: unless-stopped
    volumes:
      - ./volumes/db/13/data:/var/lib/postgresql/data
      - /etc/localtime:/etc/localtime:ro
    environment:
      - POSTGRES_USER=mmuser
      - POSTGRES_PASSWORD=mmuser_password
      - POSTGRES_DB=mattermost

  app:
    build:
      context: app
      # uncomment following lines for team edition or change UID/GID
      args:
        - edition=team
        - PUID=2179
        - PGID=2000
    ports:
      - "8443:8443/udp"
    restart: unless-stopped
    volumes:
      - ./volumes/app/mattermost/config:/mattermost/config:rw
      - ./volumes/app/mattermost/data:/mattermost/data:rw
      - ./volumes/app/mattermost/logs:/mattermost/logs:rw
      - ./volumes/app/mattermost/bleveindexes:/mattermost/bleveindexes:rw
      - ./volumes/app/mattermost/plugins:/mattermost/plugins:rw
      - ./volumes/app/mattermost/client-plugins:/mattermost/client/plugins:rw
      - /etc/localtime:/etc/localtime:ro
      # When you want to use SSO with GitLab, you have to add the cert pki chain of GitLab inside Alpine
      # to avoid Token request failed: certificate signed by unknown authority (link: https://github.com/mattermost/mattermost-server/issues/13059)
      # - <path_to_your_gitlab_pki>/pki_chain.pem:/etc/ssl/certs/pki_chain.pem:ro
    environment:
      # set same as db credentials and dbname
      - MM_USERNAME=mmuser
      - MM_PASSWORD=mmuser_password
      - MM_DBNAME=mattermost
      # For websocket
      - MM_SERVICESETTINGS_ENABLERELIABLEWEBSOCKETS=true

      # use the credentials you've set above, in the format:
      # MM_SQLSETTINGS_DATASOURCE=postgres://${MM_USERNAME}:${MM_PASSWORD}@db:5432/${MM_DBNAME}?sslmode=disable&connect_timeout=10
      - MM_SQLSETTINGS_DATASOURCE=postgres://mmuser:mmuser_password@db:5432/mattermost?sslmode=disable&connect_timeout=10

  web:
    build: web
    ports:
      - "8065:8080"
    read_only: true
    restart: unless-stopped
    volumes:
      # This directory must have cert files if you want to enable SSL
      - ./volumes/web/cert:/cert:ro
      - /etc/localtime:/etc/localtime:ro
    cap_drop:
      - ALL
2

Hi @gabrielctn ,

to me this does not look related to any changes with regards to iptables. The first errors we need to take care of are these from your application container:

app_1  | error [2023-02-03 10:38:29.949 +01:00] Failed to sync plugin from file store         caller="app/plugin.go:366" bundle=plugins/channel-notes.tar.gz error="installExtractedPlugin: Unable to move plugin from temporary directory to final destination. Another plugin may be using the same directory name., destination already exists"
app_1  | error [2023-02-03 10:38:30.723 +01:00] Failed to sync plugin from file store         caller="app/plugin.go:366" bundle=plugins/com.github.matterpoll.matterpoll.tar.gz error="installExtractedPlugin: Unable to move plugin from temporary directory to final destination. Another plugin may be using the same directory name., destination already exists"

This usually indicates write permissions on the files on the filesystem.
You seem to be using custom UID/GIDs in your docker-compose file for the application container, is that by intention and if so, can you please verify that all files below the ./volumes/app folder have the correct ownership (uid 2179 and gid 2000)?
If you want to search for files not owned by UID 2179 or group 2000, you can use this command:

find ./volumes ! -user 2179
find ./volumes ! -group 2000

You should then change them or analyze why they’re wrong first.
If you don’t care and just want to forcefully change them all, run:

chown -R 2179:2000 ./volumes

I assume that the new focalboard plugin version has already been installed some time ago and also has migrated the database successfully to the new schema and you’re now starting the old focalboard plugin version (because the system cannot install the new version over the old one due to permission issues) and this one is not compatible with the new DB schema, which might be the reason for this error in the app logs:

app_1  | error [2023-02-03 10:38:32.222 +01:00] Table creation / migration failed             caller="app/plugin_api.go:976" plugin_id=focalboard error=""pq: column "dirty" does not exist in line 0: SELECT version, dirty FROM "public"."focalboard_schema_migrations" LIMIT 1""

Your DB logs show the same error, but additionally they also tell us about the remindbot Plugin user which cannot be created, which could be OK if it’s already there and if you’re still using the plugin. Not sure if this plugin is properly coded to create the user only if it does not exist or if it just tries to create it and assumes that the database will handle the deduplication (which it does as you can see):

db_1   | 2023-02-03 09:38:30.548 UTC [44] ERROR:  duplicate key value violates unique constraint "users_email_key"
db_1   | 2023-02-03 09:38:30.548 UTC [44] DETAIL:  Key (email)=(remindbot@localhost) already exists.
db_1   | 2023-02-03 09:38:30.548 UTC [44] STATEMENT:  INSERT INTO Users
db_1   | 			(Id, CreateAt, UpdateAt, DeleteAt, Username, Password, AuthData, AuthService,
db_1   | 				Email, EmailVerified, Nickname, FirstName, LastName, Position, Roles, AllowMarketing,
db_1   | 				Props, NotifyProps, LastPasswordUpdate, LastPictureUpdate, FailedAttempts,
db_1   | 				Locale, Timezone, MfaActive, MfaSecret, RemoteId)
db_1   | 			VALUES
db_1   | 			($1, $2, $3, $4, $5, $6, $7, $8,
db_1   | 				$9, $10, $11, $12, $13, $14, $15, $16,
db_1   | 				$17, $18, $19, $20, $21,
db_1   | 				$22, $23, $24, $25, $26)

Are you using the remindbot plugin and if so, in what version and is it working?

3

Thanks a lot for your answer and help @agriesser

I do use custom UID and GID because mattermost asks to use a non-root user at installation time so I chose these (my normal uid user with local group).

I ran the command to change the permissions:

$ chown -R 2179:2000 ./volumes
$ ls -l ./volumes
drwxr-xr-x 3 2179 2000 4096 févr. 16  2021 app
drwxr-xr-x 3 2179 2000 4096 août  31  2021 db
drwxr-xr-x 3 2179 2000 4096 févr. 16  2021 web

I also disabled the remind and channel-notes plugins in the conf file (we didn’t really use them anyway):

"PluginSettings": {
    "PluginStates": {
        "channel-notes": {
                "Enable": false
            },
        "com.github.scottleedavis.mattermost-plugin-remind": {
                "Enable": false
            }
    }
}

When I launch the docker I still get the Connection refused error, but it seems like some errors went away. However, still the issue in the database about the dirty column…

APP

$ docker-compose logs -f app
Attaching to mattermost_app_1
app_1  | Using existing config file /mattermost/config/config.json
app_1  | Using existing database connection
app_1  | Starting mattermost
app_1  | info  [2023-02-06 11:49:54.003 +01:00] Server is initializing...                     caller="platform/service.go:165" go_version=go1.18.1
app_1  | info  [2023-02-06 11:49:54.003 +01:00] EXPERIMENTAL: Starting Bleve                  caller="bleveengine/bleve.go:185"
app_1  | info  [2023-02-06 11:49:54.027 +01:00] Pinging SQL                                   caller="sqlstore/store.go:238" database=master
app_1  | info  [2023-02-06 11:49:54.220 +01:00] Starting websocket hubs                       caller="platform/web_hub.go:95" number_of_hubs=16
app_1  | info  [2023-02-06 11:49:54.240 +01:00] Loaded system translations                    caller="i18n/i18n.go:93" for locale=fr from locale=/mattermost/i18n/fr.json
app_1  | info  [2023-02-06 11:49:54.254 +01:00] Current version is 7.7.1 (7.7.1/Thu Jan 19 00:47:45 UTC 2023/63149c4869898f8c1d3a94ea728370ec4a07575f/none) caller="app/server.go:395" current_version=7.7.1 build_number=7.7.1 build_date="Thu Jan 19 00:47:45 UTC 2023" build_hash=63149c4869898f8c1d3a94ea728370ec4a07575f build_hash_enterprise=none
app_1  | info  [2023-02-06 11:49:54.254 +01:00] Team Edition Build                            caller="app/server.go:406" enterprise_build=false
app_1  | info  [2023-02-06 11:49:54.254 +01:00] Printing current working                      caller="app/server.go:410" directory=/mattermost
app_1  | info  [2023-02-06 11:49:54.254 +01:00] Loaded config                                 caller="app/server.go:411" source="file:///mattermost/config/config.json"
app_1  | info  [2023-02-06 11:49:54.260 +01:00] Starting workers                              caller="jobs/workers.go:48"
app_1  | info  [2023-02-06 11:49:54.261 +01:00] Starting schedulers.                          caller="jobs/schedulers.go:47"
app_1  | error [2023-02-06 11:49:54.277 +01:00] License key from https://mattermost.com required to unlock enterprise features. caller="platform/license.go:106" error="resource: License id: "
app_1  | info  [2023-02-06 11:49:54.575 +01:00] Starting up plugins                           caller="app/plugin.go:213"
app_1  | info  [2023-02-06 11:49:54.575 +01:00] Syncing plugins from the file store           caller="app/plugin.go:294"
app_1  | info  [2023-02-06 11:49:54.611 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.github.scottleedavis.mattermost-plugin-remind.tar.gz
app_1  | info  [2023-02-06 11:49:54.611 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.github.matterpoll.matterpoll.tar.gz
app_1  | info  [2023-02-06 11:49:54.611 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.github.moussetc.mattermost.plugin.giphy.tar.gz
app_1  | info  [2023-02-06 11:49:54.611 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.mattermost.calls.tar.gz
app_1  | info  [2023-02-06 11:49:54.611 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.matthewdorner.mattermost-plugin-chess.tar.gz
app_1  | info  [2023-02-06 11:49:54.611 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.mattermost.plugin-todo.tar.gz
app_1  | info  [2023-02-06 11:49:54.611 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.mattermost.draw-plugin.tar.gz
app_1  | info  [2023-02-06 11:49:54.611 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/playbooks.tar.gz
app_1  | info  [2023-02-06 11:49:54.618 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/focalboard.tar.gz
app_1  | info  [2023-02-06 11:49:54.628 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.github.moussetc.mattermost.plugin.diceroller.tar.gz
app_1  | info  [2023-02-06 11:49:54.628 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.mattermost.custom-attributes.tar.gz
app_1  | info  [2023-02-06 11:49:54.628 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.mattermost.copy-post-plugin.tar.gz
app_1  | info  [2023-02-06 11:49:54.628 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.mattermost.agenda.tar.gz
app_1  | info  [2023-02-06 11:49:54.628 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/com.mattermost.welcomebot.tar.gz
app_1  | info  [2023-02-06 11:49:54.628 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/github.tar.gz
app_1  | info  [2023-02-06 11:49:54.629 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/memes.tar.gz
app_1  | info  [2023-02-06 11:49:54.639 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/channel-notes.tar.gz
app_1  | info  [2023-02-06 11:49:54.652 +01:00] Syncing plugin from file store                caller="app/plugin.go:364" bundle=plugins/mattermost-autolink.tar.gz
app_1  | warn  [2023-02-06 11:49:56.076 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.mattermost.welcomebot
app_1  | warn  [2023-02-06 11:49:56.078 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.github.moussetc.mattermost.plugin.giphy
app_1  | warn  [2023-02-06 11:49:56.580 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=memes
app_1  | warn  [2023-02-06 11:49:56.580 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=mattermost-autolink
app_1  | warn  [2023-02-06 11:49:56.740 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.mattermost.custom-attributes
app_1  | warn  [2023-02-06 11:49:57.090 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.mattermost.agenda
app_1  | warn  [2023-02-06 11:49:57.703 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.mattermost.calls
app_1  | error [2023-02-06 11:49:57.709 +01:00] Failed to sync plugin from file store         caller="app/plugin.go:366" bundle=plugins/channel-notes.tar.gz error="installExtractedPlugin: Unable to move plugin from temporary directory to final destination. Another plugin may be using the same directory name., destination already exists"
app_1  | error [2023-02-06 11:49:58.127 +01:00] Failed to sync plugin from file store         caller="app/plugin.go:366" bundle=plugins/com.github.matterpoll.matterpoll.tar.gz error="installExtractedPlugin: Unable to move plugin from temporary directory to final destination. Another plugin may be using the same directory name., destination already exists"
app_1  | info  [2023-02-06 11:49:58.295 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:84"
app_1  | info  [2023-02-06 11:49:58.296 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:84"
app_1  | info  [2023-02-06 11:49:58.297 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:84"
app_1  | info  [2023-02-06 11:49:58.297 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:84"
app_1  | info  [2023-02-06 11:49:58.298 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:84"
app_1  | info  [2023-02-06 11:49:58.299 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:84"
app_1  | info  [2023-02-06 11:49:58.299 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:84"
app_1  | info  [2023-02-06 11:49:58.300 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:84"
app_1  | warn  [2023-02-06 11:49:58.453 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=playbooks
app_1  | warn  [2023-02-06 11:49:59.372 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=focalboard
app_1  | info  [2023-02-06 11:49:59.422 +01:00] connectDatabase                               caller="app/plugin_api.go:973" plugin_id=focalboard dbType=postgres
app_1  | error [2023-02-06 11:49:59.439 +01:00] Table creation / migration failed             caller="app/plugin_api.go:976" plugin_id=focalboard error=""pq: column "dirty" does not exist in line 0: SELECT version, dirty FROM "public"."focalboard_schema_migrations" LIMIT 1""
app_1  | info  [2023-02-06 11:49:59.442 +01:00] plugin process exited                         caller="plugin/hclog_adapter.go:61" plugin_id=focalboard wrapped_extras=path/mattermost/plugins/focalboard/server/dist/plugin-linux-amd64pid110
app_1  | error [2023-02-06 11:49:59.468 +01:00] Failed to sync plugin from file store         caller="app/plugin.go:366" bundle=plugins/focalboard.tar.gz error="installExtractedPlugin: Unable to restart plugin on upgrade., error initializing the DB: pq: column "dirty" does not exist in line 0: SELECT version, dirty FROM "public"."focalboard_schema_migrations" LIMIT 1"
app_1  | warn  [2023-02-06 11:50:00.465 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.mattermost.plugin-channel-export
app_1  | info  [2023-02-06 11:50:00.638 +01:00] rtc: server was shutdown                      caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:84"
app_1  | info  [2023-02-06 11:50:00.777 +01:00] plugin process exited                         caller="plugin/hclog_adapter.go:61" plugin_id=playbooks wrapped_extras=path/mattermost/plugins/playbooks/server/dist/plugin-linux-amd64pid97
app_1  | info  [2023-02-06 11:50:00.938 +01:00] plugin process exited                         caller="plugin/hclog_adapter.go:61" plugin_id=com.mattermost.calls wrapped_extras=path/mattermost/plugins/com.mattermost.calls/server/dist/plugin-linux-amd64pid86
app_1  | warn  [2023-02-06 11:50:01.019 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.mattermost.apps
app_1  | info  [2023-02-06 11:50:01.265 +01:00] activated                                     caller="app/plugin_api.go:973" plugin_id=com.mattermost.apps
app_1  | warn  [2023-02-06 11:50:02.680 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=playbooks
app_1  | info  [2023-02-06 11:50:02.903 +01:00] registered collection and topic type          caller="app/collection.go:33" plugin_id=playbooks collection_type=run topic_type=status
app_1  | info  [2023-02-06 11:50:02.903 +01:00] registered collection and topic type          caller="app/collection.go:33" plugin_id=playbooks collection_type=run topic_type=task
app_1  | warn  [2023-02-06 11:50:03.060 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.mattermost.calls
app_1  | info  [2023-02-06 11:50:03.453 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:90"
app_1  | info  [2023-02-06 11:50:03.453 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:90"
app_1  | info  [2023-02-06 11:50:03.454 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:90"
app_1  | info  [2023-02-06 11:50:03.455 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:90"
app_1  | info  [2023-02-06 11:50:03.456 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:90"
app_1  | info  [2023-02-06 11:50:03.457 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:90"
app_1  | info  [2023-02-06 11:50:03.457 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:90"
app_1  | info  [2023-02-06 11:50:03.459 +01:00] rtc: server is listening on udp 8443          caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:90"
app_1  | warn  [2023-02-06 11:50:03.507 +01:00] plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=focalboard
app_1  | info  [2023-02-06 11:50:03.683 +01:00] connectDatabase                               caller="app/plugin_api.go:973" plugin_id=focalboard dbType=postgres
app_1  | info  [2023-02-06 11:50:03.705 +01:00] {"level":"info","msg":"Pinging SQL","fields":{"database":"master"}}
app_1  |  caller="io/io.go:428" plugin_id=focalboard source=plugin_stderr
app_1  | info  [2023-02-06 11:50:03.819 +01:00] Initialized notification backend              caller="app/plugin_api.go:973" plugin_id=focalboard name=notifyMentions
app_1  | info  [2023-02-06 11:50:03.820 +01:00] Initialized notification backend              caller="app/plugin_api.go:973" plugin_id=focalboard name=notifySubscriptions
app_1  | info  [2023-02-06 11:50:03.820 +01:00] Initialized notification backend              caller="app/plugin_api.go:973" plugin_id=focalboard name=notifyLogger
app_1  | info  [2023-02-06 11:50:03.836 +01:00] FocalBoard Server                             caller="app/plugin_api.go:973" plugin_id=focalboard version=7.7.0 edition=plugin build_number=3856523067 build_date=""Fri Jan  6 15:51:58 UTC 2023"" build_hash=938e7f1c82c7af00115248e284f302c28557f040
app_1  | info  [2023-02-06 11:50:03.836 +01:00] Server.Start                                  caller="app/plugin_api.go:973" plugin_id=focalboard
app_1  | info  [2023-02-06 11:50:03.851 +01:00] Boards product successfully started.          caller="app/plugin_api.go:973" plugin_id=focalboard
app_1  | info  [2023-02-06 11:50:03.899 +01:00] Starting Server...                            caller="app/server.go:878"
app_1  | info  [2023-02-06 11:50:03.899 +01:00] RateLimiter is enabled                        caller="app/server.go:912"
app_1  | info  [2023-02-06 11:50:03.899 +01:00] Server is listening on [::]:8065              caller="app/server.go:950" address="[::]:8065"

DB

$ docker-compose logs -f db
Attaching to mattermost_db_1
db_1   | AWS_ACCESS_KEY_ID is required for Wal-E but not set. Skipping Wal-E setup.
db_1   | AWS_SECRET_ACCESS_KEY is required for Wal-E but not set. Skipping Wal-E setup.
db_1   | WALE_S3_PREFIX is required for Wal-E but not set. Skipping Wal-E setup.
db_1   | AWS_REGION is required for Wal-E but not set. Skipping Wal-E setup.
db_1   |
db_1   | PostgreSQL Database directory appears to contain a database; Skipping initialization
db_1   |
db_1   | 2023-02-06 10:49:52.310 UTC [14] LOG:  starting PostgreSQL 13.2 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.2.1_pre1) 10.2.1 20201203, 64-bit
db_1   | 2023-02-06 10:49:52.310 UTC [14] LOG:  listening on IPv4 address "0.0.0.0", port 5432
db_1   | 2023-02-06 10:49:52.310 UTC [14] LOG:  listening on IPv6 address "::", port 5432
db_1   | 2023-02-06 10:49:52.338 UTC [14] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1   | 2023-02-06 10:49:52.343 UTC [29] LOG:  database system was interrupted; last known up at 2023-02-06 10:42:16 UTC
db_1   | 2023-02-06 10:49:52.383 UTC [29] LOG:  database system was not properly shut down; automatic recovery in progress
db_1   | 2023-02-06 10:49:52.386 UTC [29] LOG:  redo starts at 8/BEB8CA8
db_1   | 2023-02-06 10:49:52.386 UTC [29] LOG:  invalid record length at 8/BEBD698: wanted 24, got 0
db_1   | 2023-02-06 10:49:52.386 UTC [29] LOG:  redo done at 8/BEBD660
db_1   | 2023-02-06 10:49:52.421 UTC [14] LOG:  database system is ready to accept connections
db_1   | 2023-02-06 10:49:59.437 UTC [46] ERROR:  column "dirty" does not exist at character 17
db_1   | 2023-02-06 10:49:59.437 UTC [46] STATEMENT:  SELECT version, dirty FROM "public"."focalboard_schema_migrations" LIMIT 1

WEB

$ docker-compose logs -f web
Attaching to mattermost_web_1
web_1  | linking plain config
web_1  | 2023/02/06 11:17:15 [error] 18#18: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.27.0.1, server: , request: "GET /api/v4/websocket?connection_id=uaeup8iza3dctyyoanwsnni6wh&sequence_number=479 HTTP/1.1", upstream: "http://172.27.0.2:8000/api/v4/websocket?connection_id=uaeup8iza3dctyyoanwsnni6wh&sequence_number=479", host: "mattermost.dsimb.inserm.fr"
web_1  | 172.27.0.1 - - [06/Feb/2023:11:17:15 +0100] "GET /api/v4/websocket?connection_id=uaeup8iza3dctyyoanwsnni6wh&sequence_number=479 HTTP/1.1" 502 552 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.5249.119 Electron/21.2.0 Safari/537.36 Mattermost/5.2.2" "10.93.167.136"
web_1  | 2023/02/06 11:17:15 [error] 18#18: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 172.27.0.1, server: , request: "GET /api/v4/websocket?connection_id=icwmjqo6ftb9m81jks68k5xemo&sequence_number=4461 HTTP/1.1", upstream: "http://172.27.0.2:8000/api/v4/websocket?connection_id=icwmjqo6ftb9m81jks68k5xemo&sequence_number=4461", host: "mattermost.dsimb.inserm.fr"
4

I’ve set the log level to DEBUG to see more information but the log of APP is too long to paste here.
So here is truncated parts which I find are the most relevant:

APP

$ docker-compose logs -f app
Attaching to mattermost_app_1
warn  plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.mattermost.welcomebot
debug starting plugin                               caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.welcomebot wrapped_extras="path/mattermost/plugins/com.mattermost.welcomebot/server/dist/plugin-linux-amd64args[/mattermost/plugins/com.mattermost.welcomebot/server/dist/plugin-linux-amd64]"
debug plugin started                                caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.welcomebot wrapped_extras=path/mattermost/plugins/com.mattermost.welcomebot/server/dist/plugin-linux-amd64pid36
debug waiting for RPC address                       caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.welcomebot wrapped_extras=path/mattermost/plugins/com.mattermost.welcomebot/server/dist/plugin-linux-amd64
debug plugin address                                caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.welcomebot wrapped_extras="networkunixaddress/tmp/plugin642524629timestamp2023-02-06T11:17:18.188+0100"
debug using plugin                                  caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.welcomebot wrapped_extras=version1
warn  plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.mattermost.custom-attributes
debug starting plugin                               caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.custom-attributes wrapped_extras="path/mattermost/plugins/com.mattermost.custom-attributes/server/dist/plugin-linux-amd64args[/mattermost/plugins/com.mattermost.custom-attributes/server/dist/plugin-linux-amd64]"
debug plugin started                                caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.custom-attributes wrapped_extras=path/mattermost/plugins/com.mattermost.custom-attributes/server/dist/plugin-linux-amd64pid44
debug waiting for RPC address                       caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.custom-attributes wrapped_extras=path/mattermost/plugins/com.mattermost.custom-attributes/server/dist/plugin-linux-amd64
debug plugin address                                caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.custom-attributes wrapped_extras="networkunixaddress/tmp/plugin2327608007timestamp2023-02-06T11:17:18.354+0100"
debug using plugin                                  caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.custom-attributes wrapped_extras=version1
warn  plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.github.moussetc.mattermost.plugin.giphy
debug starting plugin                               caller="plugin/hclog_adapter.go:52" plugin_id=com.github.moussetc.mattermost.plugin.giphy wrapped_extras="path/mattermost/plugins/com.github.moussetc.mattermost.plugin.giphy/server/dist/plugin-linux-amd64args[/mattermost/plugins/com.github.moussetc.mattermost.plugin.giphy/server/dist/plugin-linux-amd64]"
debug plugin started                                caller="plugin/hclog_adapter.go:52" plugin_id=com.github.moussetc.mattermost.plugin.giphy wrapped_extras=path/mattermost/plugins/com.github.moussetc.mattermost.plugin.giphy/server/dist/plugin-linux-amd64pid51
debug waiting for RPC address                       caller="plugin/hclog_adapter.go:52" plugin_id=com.github.moussetc.mattermost.plugin.giphy wrapped_extras=path/mattermost/plugins/com.github.moussetc.mattermost.plugin.giphy/server/dist/plugin-linux-amd64
warn  plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=memes
debug starting plugin                               caller="plugin/hclog_adapter.go:52" plugin_id=memes wrapped_extras="path/mattermost/plugins/memes/server/dist/plugin-linux-amd64args[/mattermost/plugins/memes/server/dist/plugin-linux-amd64]"
debug plugin address                                caller="plugin/hclog_adapter.go:52" plugin_id=com.github.moussetc.mattermost.plugin.giphy wrapped_extras="address/tmp/plugin825010832networkunixtimestamp2023-02-06T11:17:18.476+0100"
debug plugin started                                caller="plugin/hclog_adapter.go:52" plugin_id=memes wrapped_extras=path/mattermost/plugins/memes/server/dist/plugin-linux-amd64pid57
debug waiting for RPC address                       caller="plugin/hclog_adapter.go:52" plugin_id=memes wrapped_extras=path/mattermost/plugins/memes/server/dist/plugin-linux-amd64
debug using plugin                                  caller="plugin/hclog_adapter.go:52" plugin_id=com.github.moussetc.mattermost.plugin.giphy wrapped_extras=version1
warn  plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=mattermost-autolink
debug starting plugin                               caller="plugin/hclog_adapter.go:52" plugin_id=mattermost-autolink wrapped_extras="path/mattermost/plugins/mattermost-autolink/server/dist/plugin-linux-amd64args[/mattermost/plugins/mattermost-autolink/server/dist/plugin-linux-amd64]"
debug plugin started                                caller="plugin/hclog_adapter.go:52" plugin_id=mattermost-autolink wrapped_extras=path/mattermost/plugins/mattermost-autolink/server/dist/plugin-linux-amd64pid69
debug waiting for RPC address                       caller="plugin/hclog_adapter.go:52" plugin_id=mattermost-autolink wrapped_extras=path/mattermost/plugins/mattermost-autolink/server/dist/plugin-linux-amd64
debug plugin address                                caller="plugin/hclog_adapter.go:52" plugin_id=mattermost-autolink wrapped_extras="networkunixaddress/tmp/plugin320241053timestamp2023-02-06T11:17:19.214+0100"
debug using plugin                                  caller="plugin/hclog_adapter.go:52" plugin_id=mattermost-autolink wrapped_extras=version1
debug plugin address                                caller="plugin/hclog_adapter.go:52" plugin_id=memes wrapped_extras="address/tmp/plugin452248626networkunixtimestamp2023-02-06T11:17:19.262+0100"
debug using plugin                                  caller="plugin/hclog_adapter.go:52" plugin_id=memes wrapped_extras=version1
warn  plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.mattermost.agenda
debug starting plugin                               caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.agenda wrapped_extras="path/mattermost/plugins/com.mattermost.agenda/server/dist/plugin-linux-amd64args[/mattermost/plugins/com.mattermost.agenda/server/dist/plugin-linux-amd64]"
debug plugin started                                caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.agenda wrapped_extras=path/mattermost/plugins/com.mattermost.agenda/server/dist/plugin-linux-amd64pid81
debug waiting for RPC address                       caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.agenda wrapped_extras=path/mattermost/plugins/com.mattermost.agenda/server/dist/plugin-linux-amd64
debug plugin address                                caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.agenda wrapped_extras="address/tmp/plugin948626481networkunixtimestamp2023-02-06T11:17:19.392+0100"
debug using plugin                                  caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.agenda wrapped_extras=version1
error Failed to sync plugin from file store         caller="app/plugin.go:366" bundle=plugins/channel-notes.tar.gz error="installExtractedPlugin: Unable to move plugin from temporary directory to final destination. Another plugin may be using the same directory name., destination already exists"
warn  plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.mattermost.calls
debug starting plugin                               caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.calls wrapped_extras="path/mattermost/plugins/com.mattermost.calls/server/dist/plugin-linux-amd64args[/mattermost/plugins/com.mattermost.calls/server/dist/plugin-linux-amd64]"
debug plugin started                                caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.calls wrapped_extras=path/mattermost/plugins/com.mattermost.calls/server/dist/plugin-linux-amd64pid89
debug waiting for RPC address                       caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.calls wrapped_extras=path/mattermost/plugins/com.mattermost.calls/server/dist/plugin-linux-amd64
error Failed to sync plugin from file store         caller="app/plugin.go:366" bundle=plugins/com.github.matterpoll.matterpoll.tar.gz error="installExtractedPlugin: Unable to move plugin from temporary directory to final destination. Another plugin may be using the same directory name., destination already exists"
debug plugin address                                caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.calls wrapped_extras="address/tmp/plugin3396022555networkunixtimestamp2023-02-06T11:17:20.568+0100"
debug using plugin                                  caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.calls wrapped_extras=version1
debug Initializing telemetry                        caller="app/plugin_api.go:970" plugin_id=com.mattermost.calls origin="main.(*Plugin).initTelemetry telemetry.go:45"
debug activating                                    caller="app/plugin_api.go:970" plugin_id=com.mattermost.calls origin="main.(*Plugin).OnActivate activate.go:19"
debug cleaning up calls state                       caller="app/plugin_api.go:970" plugin_id=com.mattermost.calls origin="main.(*Plugin).cleanUpState channel_state.go:89"
warn  plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=playbooks
debug starting plugin                               caller="plugin/hclog_adapter.go:52" plugin_id=playbooks wrapped_extras="path/mattermost/plugins/playbooks/server/dist/plugin-linux-amd64args[/mattermost/plugins/playbooks/server/dist/plugin-linux-amd64]"
debug plugin started                                caller="plugin/hclog_adapter.go:52" plugin_id=playbooks wrapped_extras=path/mattermost/plugins/playbooks/server/dist/plugin-linux-amd64pid100
debug waiting for RPC address                       caller="plugin/hclog_adapter.go:52" plugin_id=playbooks wrapped_extras=path/mattermost/plugins/playbooks/server/dist/plugin-linux-amd64
debug using plugin                                  caller="plugin/hclog_adapter.go:52" plugin_id=playbooks wrapped_extras=version1
debug plugin address                                caller="plugin/hclog_adapter.go:52" plugin_id=playbooks wrapped_extras="address/tmp/plugin302709157networkunixtimestamp2023-02-06T11:17:21.286+0100"
warn  plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=focalboard
debug starting plugin                               caller="plugin/hclog_adapter.go:52" plugin_id=focalboard wrapped_extras="path/mattermost/plugins/focalboard/server/dist/plugin-linux-amd64args[/mattermost/plugins/focalboard/server/dist/plugin-linux-amd64]"
debug plugin started                                caller="plugin/hclog_adapter.go:52" plugin_id=focalboard wrapped_extras=path/mattermost/plugins/focalboard/server/dist/plugin-linux-amd64pid112
debug waiting for RPC address                       caller="plugin/hclog_adapter.go:52" plugin_id=focalboard wrapped_extras=path/mattermost/plugins/focalboard/server/dist/plugin-linux-amd64
debug using plugin                                  caller="plugin/hclog_adapter.go:52" plugin_id=focalboard wrapped_extras=version1
debug plugin address                                caller="plugin/hclog_adapter.go:52" plugin_id=focalboard wrapped_extras="address/tmp/plugin348748700networkunixtimestamp2023-02-06T11:17:22.208+0100"
info  connectDatabase                               caller="app/plugin_api.go:973" plugin_id=focalboard dbType=postgres
error Table creation / migration failed             caller="app/plugin_api.go:976" plugin_id=focalboard error=""pq: column "dirty" does not exist in line 0: SELECT version, dirty FROM "public"."focalboard_schema_migrations" LIMIT 1""
info  plugin process exited                         caller="plugin/hclog_adapter.go:61" plugin_id=focalboard wrapped_extras=path/mattermost/plugins/focalboard/server/dist/plugin-linux-amd64pid112
debug plugin exited                                 caller="plugin/hclog_adapter.go:54" plugin_id=focalboard
error Failed to sync plugin from file store         caller="app/plugin.go:366" bundle=plugins/focalboard.tar.gz error="installExtractedPlugin: Unable to restart plugin on upgrade., error initializing the DB: pq: column "dirty" does not exist in line 0: SELECT version, dirty FROM "public"."focalboard_schema_migrations" LIMIT 1"
debug Processing prepackaged plugin                 caller="app/plugin.go:987" path=/mattermost/prepackaged_plugins/mattermost-plugin-custom-attributes-v1.3.1-linux-amd64.tar.gz
debug Processing prepackaged plugin                 caller="app/plugin.go:987" path=/mattermost/prepackaged_plugins/focalboard-v7.7.0-linux-amd64.tar.gz
debug Processing prepackaged plugin                 caller="app/plugin.go:987" path=/mattermost/prepackaged_plugins/mattermost-plugin-nps-v1.3.1-linux-amd64.tar.gz
debug Processing prepackaged plugin                 caller="app/plugin.go:987" path=/mattermost/prepackaged_plugins/mattermost-plugin-zoom-v1.6.0-linux-amd64.tar.gz
debug Processing prepackaged plugin                 caller="app/plugin.go:987" path=/mattermost/prepackaged_plugins/mattermost-plugin-antivirus-v0.1.2-linux-amd64.tar.gz
debug Processing prepackaged plugin                 caller="app/plugin.go:987" path=/mattermost/prepackaged_plugins/mattermost-plugin-jira-v3.2.2-linux-amd64.tar.gz
debug Processing prepackaged plugin                 caller="app/plugin.go:987" path=/mattermost/prepackaged_plugins/mattermost-plugin-jitsi-v2.0.1-linux-amd64.tar.gz
debug Processing prepackaged plugin                 caller="app/plugin.go:987" path=/mattermost/prepackaged_plugins/mattermost-plugin-autolink-v1.2.2-linux-amd64.tar.gz
debug Processing prepackaged plugin                 caller="app/plugin.go:987" path=/mattermost/prepackaged_plugins/mattermost-plugin-jenkins-v1.1.0-linux-amd64.tar.gz
debug Processing prepackaged plugin                 caller="app/plugin.go:987" path=/mattermost/prepackaged_plugins/mattermost-plugin-apps-v1.1.0-linux-amd64.tar.gz
debug Processing prepackaged plugin                 caller="app/plugin.go:987" path=/mattermost/prepackaged_plugins/mattermost-plugin-todo-v0.6.1-linux-amd64.tar.gz
debug Processing prepackaged plugin                 caller="app/plugin.go:987" path=/mattermost/prepackaged_plugins/mattermost-plugin-playbooks-v1.35.0-linux-amd64.tar.gz
debug Processing prepackaged plugin                 caller="app/plugin.go:987" path=/mattermost/prepackaged_plugins/mattermost-plugin-aws-SNS-v1.2.0-linux-amd64.tar.gz
debug Processing prepackaged plugin                 caller="app/plugin.go:987" path=/mattermost/prepackaged_plugins/mattermost-plugin-gitlab-v1.6.0-linux-amd64.tar.gz
debug Processing prepackaged plugin                 caller="app/plugin.go:987" path=/mattermost/prepackaged_plugins/mattermost-plugin-channel-export-v1.0.0-linux-amd64.tar.gz
debug Processing prepackaged plugin                 caller="app/plugin.go:987" path=/mattermost/prepackaged_plugins/mattermost-plugin-welcomebot-v1.2.0-linux-amd64.tar.gz
debug Processing prepackaged plugin                 caller="app/plugin.go:987" path=/mattermost/prepackaged_plugins/mattermost-plugin-calls-v0.12.1-linux-amd64.tar.gz
debug Processing prepackaged plugin                 caller="app/plugin.go:987" path=/mattermost/prepackaged_plugins/mattermost-plugin-github-v2.1.4-linux-amd64.tar.gz
debug Processing prepackaged plugin                 caller="app/plugin.go:987" path=/mattermost/prepackaged_plugins/mattermost-plugin-confluence-v1.3.0-linux-amd64.tar.gz
debug Installing prepackaged plugin                 caller="app/plugin.go:1017" path=/mattermost/prepackaged_plugins/mattermost-plugin-autolink-v1.2.2-linux-amd64.tar.gz
debug Skipping local installation of plugin since existing version is newer caller="app/plugin_install.go:356" plugin_id=mattermost-autolink
debug Installing prepackaged plugin                 caller="app/plugin.go:1017" path=/mattermost/prepackaged_plugins/mattermost-plugin-welcomebot-v1.2.0-linux-amd64.tar.gz
debug Skipping local installation of plugin since existing version is newer caller="app/plugin_install.go:356" plugin_id=com.mattermost.welcomebot
debug Installing prepackaged plugin                 caller="app/plugin.go:1017" path=/mattermost/prepackaged_plugins/mattermost-plugin-custom-attributes-v1.3.1-linux-amd64.tar.gz
debug Skipping local installation of plugin since existing version is newer caller="app/plugin_install.go:356" plugin_id=com.mattermost.custom-attributes
debug Installing prepackaged plugin                 caller="app/plugin.go:1017" path=/mattermost/prepackaged_plugins/mattermost-plugin-channel-export-v1.0.0-linux-amd64.tar.gz
debug Installing prepackaged plugin                 caller="app/plugin.go:1017" path=/mattermost/prepackaged_plugins/mattermost-plugin-apps-v1.1.0-linux-amd64.tar.gz
debug Installing prepackaged plugin                 caller="app/plugin.go:1017" path=/mattermost/prepackaged_plugins/mattermost-plugin-calls-v0.12.1-linux-amd64.tar.gz
debug Installing prepackaged plugin                 caller="app/plugin.go:1017" path=/mattermost/prepackaged_plugins/mattermost-plugin-playbooks-v1.35.0-linux-amd64.tar.gz
debug Removing existing installation of plugin before local install caller="app/plugin_install.go:362" plugin_id=playbooks version=1.24.1
debug Removing existing installation of plugin before local install caller="app/plugin_install.go:362" plugin_id=com.mattermost.calls version=0.6.1
debug deactivate                                    caller="app/plugin_api.go:970" plugin_id=com.mattermost.calls origin="main.(*Plugin).OnDeactivate activate.go:135"
info  rtc: server was shutdown                      caller="app/plugin_api.go:973" plugin_id=com.mattermost.calls origin="main.(*logger).Info log.go:84"
info  plugin process exited                         caller="plugin/hclog_adapter.go:61" plugin_id=playbooks wrapped_extras=path/mattermost/plugins/playbooks/server/dist/plugin-linux-amd64pid100
debug plugin exited                                 caller="plugin/hclog_adapter.go:54" plugin_id=playbooks
debug cleaning up calls state                       caller="app/plugin_api.go:970" plugin_id=com.mattermost.calls origin="main.(*Plugin).cleanUpState channel_state.go:89"
warn  plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.mattermost.plugin-channel-export
debug starting plugin                               caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.plugin-channel-export wrapped_extras="path/mattermost/plugins/com.mattermost.plugin-channel-export/server/dist/plugin-linux-amd64args[/mattermost/plugins/com.mattermost.plugin-channel-export/server/dist/plugin-linux-amd64]"
debug plugin started                                caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.plugin-channel-export wrapped_extras=path/mattermost/plugins/com.mattermost.plugin-channel-export/server/dist/plugin-linux-amd64pid124
debug waiting for RPC address                       caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.plugin-channel-export wrapped_extras=path/mattermost/plugins/com.mattermost.plugin-channel-export/server/dist/plugin-linux-amd64
debug using plugin                                  caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.plugin-channel-export wrapped_extras=version1
debug plugin address                                caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.plugin-channel-export wrapped_extras="address/tmp/plugin129657622networkunixtimestamp2023-02-06T11:17:23.760+0100"
info  plugin process exited                         caller="plugin/hclog_adapter.go:61" plugin_id=com.mattermost.calls wrapped_extras=path/mattermost/plugins/com.mattermost.calls/server/dist/plugin-linux-amd64pid89
debug plugin exited                                 caller="plugin/hclog_adapter.go:54" plugin_id=com.mattermost.calls
debug Installing prepackaged plugin                 caller="app/plugin.go:1017" path=/mattermost/prepackaged_plugins/focalboard-v7.7.0-linux-amd64.tar.gz
debug Removing existing installation of plugin before local install caller="app/plugin_install.go:362" plugin_id=focalboard version=0.10.0
warn  plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.mattermost.apps
debug starting plugin                               caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.apps wrapped_extras="path/mattermost/plugins/com.mattermost.apps/server/dist/plugin-linux-amd64args[/mattermost/plugins/com.mattermost.apps/server/dist/plugin-linux-amd64]"
debug plugin started                                caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.apps wrapped_extras=path/mattermost/plugins/com.mattermost.apps/server/dist/plugin-linux-amd64pid132
debug waiting for RPC address                       caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.apps wrapped_extras=path/mattermost/plugins/com.mattermost.apps/server/dist/plugin-linux-amd64
debug plugin address                                caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.apps wrapped_extras="address/tmp/plugin2628291681networkunixtimestamp2023-02-06T11:17:24.754+0100"
debug using plugin                                  caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.apps wrapped_extras=version1
debug ensured bot                                   caller="app/plugin_api.go:970" plugin_id=com.mattermost.apps username=appsbot id=cmmj4jffabrjbx1puy849zm83a
debug failed to fetch license twice. May incorrectly default to on-prem mode caller="app/plugin_api.go:970" plugin_id=com.mattermost.apps
debug configured                                    caller="app/plugin_api.go:970" plugin_id=com.mattermost.apps commit=0856d8f build_date="Tue 24 May 2022 03:36:58 PM UTC" cloud_mode=false developer_mode=false allow_http_apps=true version=1.1.0
debug created an AWS client                         caller="app/plugin_api.go:970" plugin_id=com.mattermost.apps purpose="Manifest store" region=us-east-1 access="<nil>" secret="<nil>"
debug initialized API and persistent store          caller="app/plugin_api.go:970" plugin_id=com.mattermost.apps
debug available upstream                            caller="app/plugin_api.go:970" plugin_id=com.mattermost.apps type=HTTP
debug Skipped upstream: AWS Lambda: not configured. caller="app/plugin_api.go:970" plugin_id=com.mattermost.apps error="AWS credentials are not set: not found"
debug available upstream                            caller="app/plugin_api.go:970" plugin_id=com.mattermost.apps type="Mattermost Plugin"
debug Skipped upstream: OpenFaaS: not configured.   caller="app/plugin_api.go:970" plugin_id=com.mattermost.apps error="OPENFAAS_URL environment variable must be defined: not found"
debug initialized the app proxy                     caller="app/plugin_api.go:970" plugin_id=com.mattermost.apps
info  activated                                     caller="app/plugin_api.go:973" plugin_id=com.mattermost.apps
warn  plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=playbooks
debug starting plugin                               caller="plugin/hclog_adapter.go:52" plugin_id=playbooks wrapped_extras="path/mattermost/plugins/playbooks/server/dist/plugin-linux-amd64args[/mattermost/plugins/playbooks/server/dist/plugin-linux-amd64]"
debug plugin started                                caller="plugin/hclog_adapter.go:52" plugin_id=playbooks wrapped_extras=path/mattermost/plugins/playbooks/server/dist/plugin-linux-amd64pid143
debug waiting for RPC address                       caller="plugin/hclog_adapter.go:52" plugin_id=playbooks wrapped_extras=path/mattermost/plugins/playbooks/server/dist/plugin-linux-amd64
debug using plugin                                  caller="plugin/hclog_adapter.go:52" plugin_id=playbooks wrapped_extras=version1
debug plugin address                                caller="plugin/hclog_adapter.go:52" plugin_id=playbooks wrapped_extras="networkunixaddress/tmp/plugin677225410timestamp2023-02-06T11:17:25.913+0100"
info  registered collection and topic type          caller="app/collection.go:33" plugin_id=playbooks collection_type=run topic_type=status
info  registered collection and topic type          caller="app/collection.go:33" plugin_id=playbooks collection_type=run topic_type=task
warn  plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=com.mattermost.calls
debug starting plugin                               caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.calls wrapped_extras="path/mattermost/plugins/com.mattermost.calls/server/dist/plugin-linux-amd64args[/mattermost/plugins/com.mattermost.calls/server/dist/plugin-linux-amd64]"
debug plugin started                                caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.calls wrapped_extras=path/mattermost/plugins/com.mattermost.calls/server/dist/plugin-linux-amd64pid155
debug waiting for RPC address                       caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.calls wrapped_extras=path/mattermost/plugins/com.mattermost.calls/server/dist/plugin-linux-amd64
debug using plugin                                  caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.calls wrapped_extras=version1
debug plugin address                                caller="plugin/hclog_adapter.go:52" plugin_id=com.mattermost.calls wrapped_extras="address/tmp/plugin2569535195networkunixtimestamp2023-02-06T11:17:26.300+0100"
warn  plugin configured with a nil SecureConfig     caller="plugin/hclog_adapter.go:72" plugin_id=focalboard
debug starting plugin                               caller="plugin/hclog_adapter.go:52" plugin_id=focalboard wrapped_extras="path/mattermost/plugins/focalboard/server/dist/plugin-linux-amd64args[/mattermost/plugins/focalboard/server/dist/plugin-linux-amd64]"
debug plugin started                                caller="plugin/hclog_adapter.go:52" plugin_id=focalboard wrapped_extras=path/mattermost/plugins/focalboard/server/dist/plugin-linux-amd64pid167
debug waiting for RPC address                       caller="plugin/hclog_adapter.go:52" plugin_id=focalboard wrapped_extras=path/mattermost/plugins/focalboard/server/dist/plugin-linux-amd64
debug using plugin                                  caller="plugin/hclog_adapter.go:52" plugin_id=focalboard wrapped_extras=version1
debug plugin address                                caller="plugin/hclog_adapter.go:52" plugin_id=focalboard wrapped_extras="address/tmp/plugin680402374networkunixtimestamp2023-02-06T11:17:26.782+0100"
info  connectDatabase                               caller="app/plugin_api.go:973" plugin_id=focalboard dbType=postgres
debug Acquiring cluster lock for Focalboard migrations caller="app/plugin_api.go:970" plugin_id=focalboard
debug Getting migrations connection                 caller="app/plugin_api.go:970" plugin_id=focalboard
info  {"level":"info","msg":"Pinging SQL","fields":{"database":"master"}}
 caller="io/io.go:428" plugin_id=focalboard source=plugin_stderr
debug Creating migration engine                     caller="app/plugin_api.go:970" plugin_id=focalboard
debug == Ensuring migrations applied up to version ==================== caller="app/plugin_api.go:970" plugin_id=focalboard version=14 current_version=32
debug -- There is no need of applying any migration -------------------- caller="app/plugin_api.go:970" plugin_id=focalboard
debug == Ensuring migrations applied up to version ==================== caller="app/plugin_api.go:970" plugin_id=focalboard version=18 current_version=32
debug -- There is no need of applying any migration -------------------- caller="app/plugin_api.go:970" plugin_id=focalboard
debug == Ensuring migrations applied up to version ==================== caller="app/plugin_api.go:970" plugin_id=focalboard version=20 current_version=32
debug -- There is no need of applying any migration -------------------- caller="app/plugin_api.go:970" plugin_id=focalboard
debug == Applying all remaining migrations ==================== caller="app/plugin_api.go:970" plugin_id=focalboard current_version=32
debug Closing migration engine                      caller="app/plugin_api.go:970" plugin_id=focalboard
debug Closing migrations connection                 caller="app/plugin_api.go:970" plugin_id=focalboard
debug Releasing cluster lock for Focalboard migrations caller="app/plugin_api.go:970" plugin_id=focalboard
debug Mention listener added.                       caller="app/plugin_api.go:970" plugin_id=focalboard listener_count=1
info  Initialized notification backend              caller="app/plugin_api.go:973" plugin_id=focalboard name=notifyMentions
debug Starting subscriptions backend                caller="app/plugin_api.go:970" plugin_id=focalboard freq_card=120 freq_board=86400
info  Initialized notification backend              caller="app/plugin_api.go:973" plugin_id=focalboard name=notifySubscriptions
info  Initialized notification backend              caller="app/plugin_api.go:973" plugin_id=focalboard name=notifyLogger
debug notify loop - no hints in queue               caller="app/plugin_api.go:970" plugin_id=focalboard next_check=""2023-02-06 12:17:26.939 +01:00""
debug subscription notifier loop                    caller="app/plugin_api.go:970" plugin_id=focalboard next_notify=""2023-02-06 12:17:26.939 +01:00""
debug Fetched template boards                       caller="app/plugin_api.go:970" plugin_id=focalboard count=13
debug Template import not needed, skipping          caller="app/plugin_api.go:970" plugin_id=focalboard
info  FocalBoard Server                             caller="app/plugin_api.go:973" plugin_id=focalboard version=7.7.0 edition=plugin build_number=3856523067 build_date=""Fri Jan  6 15:51:58 UTC 2023"" build_hash=938e7f1c82c7af00115248e284f302c28557f040
info  Server.Start                                  caller="app/plugin_api.go:973" plugin_id=focalboard
debug server not bind to any port                   caller="app/plugin_api.go:970" plugin_id=focalboard
info  Boards product successfully started.          caller="app/plugin_api.go:973" plugin_id=focalboard
debug Able to write files to local storage.         caller="filestore/localstore.go:77"
info  Starting Server...                            caller="app/server.go:878"
info  RateLimiter is enabled                        caller="app/server.go:912"
info  Server is listening on [::]:8065              caller="app/server.go:950" address="[::]:8065"
debug No license provided; Remote Cluster services disabled caller="app/server.go:562"
debug Scheduling Job                                caller="migrations/scheduler.go:45" scheduler=migrations
debug All migrations are complete.                  caller="migrations/scheduler.go:84" scheduler=migrations
debug Next run time for scheduler                   caller="jobs/schedulers.go:147" scheduler_name=migrations next_runtime="<nil>"
5

I think your port mappings are a bit mixed up here.
Your web container exposes the internal port 8080 as 8065, when it should take care of the ports 443 and 80 to the outside instead.

Can you share the output of docker ps so we can see the port mappings? Typically you would need to make sure that port 80 and port 443 are configured on the host level and forwarded to the web container (nginx). The app container listens on port 8065 already, so please check your nginx.conf and make sure it uses this port for the backend connection and not port 8000 as shown in the logs.

1 Like
6

Because I use an Apache reverse proxy to map the ports.
Here is the output of docker-compose ps:

      Name                  Command                State                     Ports
------------------------------------------------------------------------------------------------
mattermost_app_1   /entrypoint.sh mattermost   Up (unhealthy)   8000/tcp, 0.0.0.0:8443->8443/udp
mattermost_db_1    /entrypoint.sh postgres     Up (healthy)     5432/tcp
mattermost_web_1   /entrypoint.sh              Up (unhealthy)   0.0.0.0:8065->8080/tcp

And the apache config:

<VirtualHost *:80>
  ServerName  mattermost.dsimb.inserm.fr
  RewriteEngine on
  RewriteCond %{SERVER_NAME} =mattermost.dsimb.inserm.fr [OR]
  RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

<IfModule mod_ssl.c>
<VirtualHost *:443>
  DocumentRoot /home/www/www-tools/mattermost
  ServerName  mattermost.dsimb.inserm.fr
  ServerSignature Off

  # Protocols h2 http/1.1


  RewriteEngine On
  RewriteCond %{REQUEST_URI} /api/v[0-9]+/(users/)?websocket [NC,OR]
  RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
  RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
  RewriteRule .* ws://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]

  <Location />
      Require all granted
      ProxyPass http://127.0.0.1:8065/
      ProxyPassReverse http://127.0.0.1:8065/
      ProxyPassReverseCookieDomain 127.0.0.1 mattermost.dsimb.inserm.fr
  </Location>


  LimitRequestBody 1048576000

  ErrorLog /var/log/apache2/mattermost-error.log
  CustomLog /var/log/apache2/mattermost-access.log combined

  Include /etc/letsencrypt/options-ssl-apache.conf
  SSLCertificateFile /etc/letsencrypt/live/mattermost.dsimb.inserm.fr/cert.pem
  SSLCertificateKeyFile /etc/letsencrypt/live/mattermost.dsimb.inserm.fr/privkey.pem
  SSLCertificateChainFile /etc/letsencrypt/live/mattermost.dsimb.inserm.fr/chain.pem
</VirtualHost>
</IfModule>

<IfModule mod_headers.c>
  Header always set Strict-Transport-Security "max-age=15552000; includeSubDomains; preload"
</IfModule>

Apache log errors are full of these errors:

[proxy:error] [pid 23244] (111)Connection refused: AH00957: WS: attempt to connect to 127.0.0.1:8065 (*) failed
[Mon Feb 06 13:56:57.902182 2023] [proxy_wstunnel:error] [pid 23244] [client 10.93.167.99:52702] AH02452: failed to make connection to backend: 127.0.0.1
7

Oh my god ok that’s it it works now !!
I checked in the config file of mattermost and for unknown reason the server was configured to listen to port 8065 instead of 8000 …
Thank you so much for your help !!

Ok so now Mattermost works, that is really nice, yet I still got the error in the logs:

db_1   | 2023-02-06 13:35:20.479 UTC [47] ERROR:  column "dirty" does not exist at character 17
db_1   | 2023-02-06 13:35:20.479 UTC [47] STATEMENT:  SELECT version, dirty FROM "public"."focalboard_schema_migrations" LIMIT 1

The thing is that focalboard works, so it doesn’t seem like a critical error

8

Can you please check out this topic for the column "dirty" does not exist issue?

I’ve written detailed instructions for troubleshooting this issue there - let me know in here if you have any questions or problems applying the commands.

1 Like
9

I uninstalled and reinstalled the plugin and it works now.
I also took the time to migrate to the new image and docker repo, I think it will be fine now.
Thanks again for your help and patience :slight_smile:

10

Awesome, thanks for letting me know :slight_smile: