Websocket connect err.","error":"websocket: the client is not using the websocket protocol: 'upgrade' token not found in 'Connection

Troubleshooting
1

Hello,
I’ve installed the last stable Mattermost version 5.20.1 and I get all time this error:

{"level":"error","ts":1583512563.9900434,"caller":"api4/websocket.go:28","msg":"websocket connect err.","error":"websocket: the client is not using the websocket protocol: 'upgrade' token not found in 'Connection' header"}
{"level":"error","ts":1583512563.9902134,"caller":"mlog/log.go:175","msg":"Failed to upgrade websocket connection","path":"/api/v4/websocket","request_id":"adf43wf7ut8nuqf9b8t4oxrgbw","ip_addr":"2.229.233.238","user_id":"w91b6pe56iymtqhkd3xnd6ttno","method":"GET","err_where":"connect","http_code":500,"err_details":""}

I’m using nginx as reverse proxy and this is the configuration.

upstream backend {
   server 127.0.0.1:8065;
   keepalive 32;
}

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;

server {
   listen 80 default_server;
   server_name   mm.website.com ;
   return 301 https://$server_name$request_uri;
}

server {
   listen 443 ssl http2;
   server_name    mm.website.com;

  ssl on;
  ssl_certificate /etc/letsencrypt/live/mm.website.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/mm.website.com/privkey.pem;
  ssl_session_timeout 1d;
  ssl_protocols TLSv1.2;
  ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:50m;
  # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
  add_header Strict-Transport-Security max-age=15768000;
  # OCSP Stapling ---
  # fetch OCSP records from URL in ssl_certificate and cache them
  ssl_stapling on;
  ssl_stapling_verify on;

   location  /api/v[0-9]+/(users/)?websocket$ {
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       client_max_body_size 50M;
       #proxy_set_header Host $host;
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       client_body_timeout 60;
       send_timeout 300;
       lingering_timeout 5;
       proxy_connect_timeout 90;
       proxy_send_timeout 300;
       proxy_read_timeout 90s;
       proxy_pass http://backend;
   }

   location / {
       client_max_body_size 50M;
       proxy_set_header Connection "";
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       proxy_read_timeout 600s;
       proxy_cache mattermost_cache;
       proxy_cache_revalidate on;
       proxy_cache_min_uses 2;
       proxy_cache_use_stale timeout;
       proxy_cache_lock on;
       proxy_http_version 1.1;
       proxy_pass http://backend;
   }
}

Also there isn’t any websocket in “pending” state:

image
image2285×902 173 KB

Adding also “AllowCorsFrom”: “website.com mm.website.com:443” to config.json doesn’t help.

Could you please help me in finding the root cause of this?
Thank you.

2

Would you be open to taking a look at our troubleshooting docs related to websocket issues https://docs.mattermost.com/install/troubleshooting.html#server-administration?

3

Hello @amy.blais,
and thanks for your reply.
I read that page at least four times and tried all solutions proposed but no luck.
Tried to switched AlloCorsFrom many times with different values, also with ‘*’ but after a while error arise.
Tried also to change this line:

proxy_set_header Connection “”;

to

proxy_set_header Connection “upgrade”;

in the location / { bla bla directive , but does not help.

For sure the error comes from nginx cause it’s clearly stated in Google Chrome Developer console:

WebSocket connection to ‘wss://mm.website.com/api/v4/websocket’ failed: Error during WebSocket handshake: Unexpected response code: 400

Searching in the forum I noticed this problem has been alredy discussed various times (i.e. How to fix websocket error? - #11 by amy.blais ) but apparently without a “standard”
solution.

The only different thing from the documentation is the backend ip: I need to put 127.0.0.1 cause my virtual machine does not have an internal ip.

Thank you.

4

I have the same issue. Has this been solved?

5

Hello @fabreg and @Cobra16319

The issue is an incorrect nginx configuration. You are missing the “~” directive which turns a regex path into a prefix match which does not match a regex pattern, failing which it falls into the / block which does not set the Connection header.

This is something we have been using successfully:


   location ~ /api/v[0-9]+/(users/)?websocket$ {
           proxy_set_header Upgrade $http_upgrade;
           proxy_set_header Connection "upgrade";
           client_max_body_size 50M;
           proxy_set_header Host $http_host;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header X-Forwarded-Proto $scheme;
           proxy_set_header X-Frame-Options SAMEORIGIN;
           proxy_buffers 256 16k;
           proxy_buffer_size 16k;
           client_body_timeout 60;
           send_timeout        300;
           lingering_timeout   5;
           proxy_connect_timeout   30s;
           proxy_send_timeout      90s;
           proxy_read_timeout      90s;
           proxy_pass http://backend;
   }

   location / {
           client_max_body_size 50M;
           proxy_set_header Connection "";
           proxy_set_header Host $http_host;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header X-Forwarded-Proto $scheme;
           proxy_set_header X-Frame-Options SAMEORIGIN;
           proxy_buffers 256 16k;
           proxy_buffer_size 16k;
           proxy_connect_timeout   30s;
           proxy_read_timeout      90s;
           proxy_send_timeout      90s;
           proxy_cache mattermost_cache;
           proxy_cache_revalidate on;
           proxy_cache_min_uses 2;
           proxy_cache_use_stale timeout;
           proxy_cache_lock on;
           proxy_http_version 1.1;
           proxy_pass http://backend;
   }
2 Likes