Summary
Connection via WebSocket stays in pending state forever
Steps to reproduce
Using Mattermost 5.16.2 from GitLab Omnibus.
- Log into Mattermost
- Error message is displayed: “Please check connection, Mattermost unreachable. If issue persists, ask administrator to check WebSocket port.”
Error message only shows for some users using Firefox 71.0-1. Other Chrome and Firefox users don’t see this message, but the WebSocket fails to connect anyway. We checked that using the browser’s developer consoles. The WebSocket connect stays in “pending” state forever. The last status code shown is “101 Switching Protocols”.
Expected behavior
WebSocket connection should be fine.
Observed behavior
On the server side, we see the following error in the logs:
{"level":"error","ts":1576066796.4229283,"caller":"api4/websocket.go:27","msg":"websocket connect err.","error":"websocket: not a websocket handshake: `Sec-WebSocket-Key' header is missing or blank"}
{"level":"error","ts":1576066796.450573,"caller":"mlog/log.go:174","msg":"Failed to upgrade websocket connection","path":"/api/v4/websocket","request_id":"bab5nd5hxfym8mcjfwau1oqwje","ip_addr":"172.21.36.13","user_id":"xn8optx113ywpm4s4f35s9tg1c","method":"GET","err_where":"connect","http_code":500,"err_details":""}
I already checked the forum and the troubleshooting guide with no success. I also tested in the same network where the server is located to make sure no proxies or firewalls are in the way, but no success. This is a corporate network.
We are using GitLab Omnibus 12.5.3.
indent preformatted text by 4 spaces
Here is the auto-generated gitlab-mattermost-http.conf (replaced the real domain with “mattermost.domain”):
# This file is managed by gitlab-ctl. Manual changes will be
# erased! To change the contents below, edit /etc/gitlab/gitlab.rb
# and run `sudo gitlab-ctl reconfigure`.
## GitLab Mattermost
upstream gitlab_mattermost {
server 127.0.0.1:8065;
}
server {
listen *:80;
server_name mattermost.domain;
server_tokens off;
location / {
return 301 https://mattermost.domain:443$request_uri;
}
access_log /var/log/gitlab/nginx/gitlab_mattermost_access.log gitlab_mattermost_access;
error_log /var/log/gitlab/nginx/gitlab_mattermost_error.log;
}
server {
listen *:443 ssl http2;
server_name mattermost.domain;
server_tokens off; # don't show the version number, a security best practice
client_max_body_size 0;
ssl_certificate /etc/gitlab/ssl/mattermost.domain.crt;
ssl_certificate_key /etc/gitlab/ssl/mattermost.domain.key;
ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4';
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_session_timeout 5m;
## Real IP Module Config
## http://nginx.org/en/docs/http/ngx_http_realip_module.html
access_log /var/log/gitlab/nginx/gitlab_mattermost_access.log gitlab_mattermost_access;
error_log /var/log/gitlab/nginx/gitlab_mattermost_error.log;
location / {
## If you use HTTPS make sure you disable gzip compression
## to be safe against BREACH attack.
gzip off;
proxy_read_timeout 3600;
proxy_connect_timeout 300;
proxy_redirect off;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Forwarded-Ssl on;
proxy_pass http://gitlab_mattermost;
}
}