Wesbocket errors on Plesk and Docker

Troubleshooting
1

I installed Mattermost with docker on a server running plesk.
Mattermost itself seems to be running fine but after a few seconds I get the red websocket error on the top.
I am assuming I need to add something to the NGINX directives but I can’t figure out what I am supposed to be adding.

Currently I hav there the following

location ~* ^/api/v[0-9]+/ {
proxy_pass http://12.19.0.1:8065;

proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection “upgrade”;
proxy_set_header Host $server_name:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
proxy_set_header X-Forwarded-Proto $scheme;

proxy_read_timeout 1200s;
client_max_body_size 0;
}
location ~* ^/ {
proxy_pass http://127.0.0.1:8065;

proxy_set_header Connection “”;
proxy_set_header Host $server_name:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
proxy_set_header X-Forwarded-Proto $scheme;

proxy_hide_header X-Frame-Options;
proxy_hide_header Content-Security-Policy;
add_header X-Frame-Options “allow-from https://webmail.$server_name”;
add_header Content-Security-Policy “frame-ancestors https://webmail.$server_name”;

proxy_read_timeout 1200s;
proxy_http_version 1.1;
client_max_body_size 0;
}

(notice the different IP addresses there, One is the docker internal one and one is the local one, tried both and still nothing )
This still does not make it work.

Any help would be greatly appreciated.

Thank you

2

Hi schluchimtalk,

the shipped nginx.conf looks like this (a bit different to yours):

  location ~ /api/v[0-9]+/(users/)?websocket$ {
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
      client_max_body_size 50M;
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Frame-Options SAMEORIGIN;
      proxy_buffers 256 16k;
      proxy_buffer_size 16k;
      client_body_timeout 60;
      send_timeout 300;
      lingering_timeout 5;
      proxy_connect_timeout 90;
      proxy_send_timeout 300;
      proxy_read_timeout 90s;
      proxy_pass http://backend;
  }

  location ~ /plugins/focalboard/ws/* {
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
      client_max_body_size 50M;
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Frame-Options SAMEORIGIN;
      proxy_buffers 256 16k;
      proxy_buffer_size 16k;
      client_body_timeout 60;
      send_timeout 300;
      lingering_timeout 5;
      proxy_connect_timeout 90;
      proxy_send_timeout 300;
      proxy_read_timeout 90s;
      proxy_pass http://backend;
  }

  location / {
      client_max_body_size 50M;
      proxy_set_header Connection "";
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Frame-Options SAMEORIGIN;
      proxy_buffers 256 16k;
      proxy_buffer_size 16k;
      proxy_read_timeout 600s;
      proxy_cache mattermost_cache;
      proxy_cache_revalidate on;
      proxy_cache_min_uses 2;
      proxy_cache_use_stale timeout;
      proxy_cache_lock on;
      proxy_http_version 1.1;
      proxy_pass http://backend;
  }

Please try to incorporate the defaults into your configuration, leaving out everything related to proxy_cache. F.ex. this looks suspicious, but might be OK depending on your configuration:

Can you change that to only $http_host?

With regards to the IP addresses being used: I assume your nginx is not running inside the docker network, right? Depending on your docker configuration, 127.0.0.1:8065 might be the correct target. You can verify that by running the following command on the console of your server. If the output contains a few lines with the text Mattermost in it, you know you’re right:

# curl -s http://localhost:8065/ | grep -o Mattermost
Mattermost
Mattermost
Mattermost
Mattermost
Mattermost
Mattermost

In the default configuration, the backend is specified as backend - you can either create a configuration for that in your nginx.conf or keep using 127.0.0.1:8065 instead of it. If you decide to go for the named version, this is the config block which you can add at the top (outside the server block) of your nginx.conf:

upstream backend {
   server 127.0.0.1:8065;
   keepalive 32;
}
3

Thank you.
I tried this, however it did not work.
In my docker logs, I get the following error

{“timestamp”:“2022-11-07 21:36:11.302 -08:00”,“level”:“error”,“msg”:“Failed to upgrade websocket connection.”,“caller”:“web/context.go:117”,“path”:“/api/v4/websocket”,“request_id”:“5at4qb9i37fxubyus3bjwcdwcw”,“ip_addr”:“172.19.0.1”,“user_id”:“9c7puw3sx7yj8ckaayxia1mbzc”,“method”:“GET”,“err_where”:“connect”,“http_code”:500,“error”:“connect: Failed to upgrade websocket connection., websocket: the client is not using the websocket protocol: ‘upgrade’ token not found in ‘Connection’ header”}

{“timestamp”:“2022-11-07 21:36:11.302 -08:00”,“level”:“debug”,“msg”:“Received HTTP request”,“caller”:“web/handlers.go:165”,“method”:“GET”,“url”:“/api/v4/websocket”,“request_id”:“5at4qb9i37fxubyus3bjwcdwcw”,“status_code”:“500”}

4

What exactly did you try? How does the resulting configuration file look like now and did you make sure to reload/restart the involved services after making the changes?
The error message indicates that the sockets are not being properly forwarded to your Mattermost application server, so can you elaborate on your topology? How is the HTTP request to your Mattermost instance being handled? What’s listening on port 443 on this server and are there maybe multiple instances of nginx running in a chain (one being in a docker container, f.ex.)?

5

I guess this is a bit over my head.
I use Plesk and installed mm as a docker image.
I will paste here the main NGINX conf file contents for this domain. (previously, I added the Mattermost under additional nginx directives option that Plesk has and not in the original file)
I changed all domain references to example.com and IP address to IP. (where it has 0.0.0.0 and 127… I left alone)

#ATTENTION!

#DO NOT MODIFY THIS FILE BECAUSE IT WAS GENERATED AUTOMATICALLY,
#SO ALL YOUR CHANGES WILL BE LOST THE NEXT TIME THE FILE IS GENERATED.

server {
listen IP:443 ssl http2;

    server_name example.com;
    server_name www.example.com;
    server_name ipv4.example.com;

    ssl_certificate             /opt/psa/var/certificates/scfj9FAqg;
    ssl_certificate_key         /opt/psa/var/certificates/scfj9FAqg;

    client_max_body_size 134217728;

    access_log "/var/www/vhosts/system/example.com/logs/proxy_access_ssl_log";
    error_log "/var/www/vhosts/system/example.com/logs/proxy_error_log";

    root "/var/www/vhosts/example.com/example.com";

    #extension letsencrypt begin
    location ^~ /.well-known/acme-challenge/ {
            root /var/www/vhosts/default/htdocs;

            types { }
            default_type text/plain;

            satisfy any;
            auth_basic off;
            allow all;

            location ~ ^/\.well-known/acme-challenge.*/\. {
                    deny all;
            }
    }
    #extension letsencrypt end

    #extension sslit begin

    #extension sslit end

    #extension docker begin
    location ~ ^/.* {
            proxy_pass http://0.0.0.0:8065;
            proxy_set_header Host              $host;
            proxy_set_header X-Real-IP         $remote_addr;
            proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
    }

    #extension docker end
    #extension sectigo begin
    location ^~ /.well-known/pki-validation/fileauth.txt {
            try_files $uri $uri/ =404;
    }

    location ^~ /.well-known/pki-validation/ {
            root /var/www/vhosts/default/htdocs;

            types { }
            default_type text/plain;

            satisfy any;
            auth_basic off;
            allow all;

            location ~ ^/\.well-known/pki-validation.*/\. {
                    deny all;
            }
    }
    #extension sectigo end

    #extension mattermost begin
    # Mattermost is enabled but not configured for this domain.
    #extension mattermost end

    location / {
            proxy_read_timeout 600;
            proxy_pass https://IP:7081;
            proxy_hide_header upgrade;
            proxy_set_header Host             $host;
            proxy_set_header X-Real-IP        $remote_addr;
            proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;
            proxy_set_header X-Accel-Internal /internal-nginx-static-location;
            access_log off;

    }

    location /internal-nginx-static-location/ {
            alias /var/www/vhosts/example.com/example.com/;
            internal;
    }

    location ~ ^/(plesk-stat|awstats-icon|webstat|webstat-ssl|ftpstat|anon_ftpstat) {
            proxy_pass https://IP:7081;
            proxy_hide_header upgrade;
            proxy_set_header Host             $host;
            proxy_set_header X-Real-IP        $remote_addr;
            proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;
            proxy_set_header X-Accel-Internal /internal-nginx-static-location;
            access_log off;

    }

    disable_symlinks if_not_owner "from=/var/www/vhosts/example.com";

    add_header X-Powered-By PleskLin;

    include "/var/www/vhosts/system/exapmle.com/conf/vhost_nginx.conf";

}
server {
listen IP:80;

    server_name example.com;
    server_name www.example.com;
    server_name ipv4.example.com;

    client_max_body_size 134217728;

    access_log "/var/www/vhosts/system/example.com/logs/proxy_access_log";
    error_log "/var/www/vhosts/system/example.com/logs/proxy_error_log";

    location / {
            return 301 https://$host$request_uri;
    }

}

6

OK, so the rest of the Mattermost specific configuration is in the file /var/www/vhosts/system/exapmle.com/conf/vhost_nginx.conf?

In this nginx config, it clearly states:

    #extension mattermost begin
    # Mattermost is enabled but not configured for this domain.
    #extension mattermost end

So not sure what to do here since I’ve never tried to install it on Plesk, but maybe you will need to activate the extension for this domain so it configures the nginx vhost properly?

The extension docker has been enabled as it seems and it added the following configuration block:

    #extension docker begin
    location ~ ^/.* {
            proxy_pass http://0.0.0.0:8065;
            proxy_set_header Host              $host;
            proxy_set_header X-Real-IP         $remote_addr;
            proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
    }

    #extension docker end

This is basically correct, but not enough for Mattermost since it’s missing the websocket specific configuration here, so you could try to disable the extension docker for this VHOST and enable the extension mattermost. Let’s see what the resulting config file looks like then.

Also: You might also want to disable one of the SSL extensions, you will most likely not need sectigo and letsencrypt active at the same time in the same VHOST.