Nginx proxy config - websocket connections get 403

Troubleshooting
1

Hi. I’m in the process of configuring nginx proxy for mattermost, and can’t seem to understand what causes websocket connections to 403. Note the official nginx config published here works just fine. It’s just that I’d like to use the common proxy configuration that’s already defined on the server.

Conf:
mattermost site:

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;

server {
    listen 443 ssl http2;
    ssl on;
    server_name mattermost.example.com;

    location / {
        include /config/nginx/proxy.conf;
        proxy_pass http://192.168.1.100:8065;

        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        proxy_cache mattermost_cache;
        proxy_cache_revalidate on;
        proxy_cache_min_uses 2;
        proxy_cache_use_stale timeout;
        proxy_cache_lock on;

        access_log      /config/log/nginx/mattermost.access.log;
        error_log       /config/log/nginx/mattermost.error.log;
    }

    location /api/v3/users/websocket {
        include /config/nginx/proxy.conf;
        proxy_pass http://192.168.1.100:8065;

        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        access_log      /config/log/nginx/mattermost.access.log;
        error_log       /config/log/nginx/mattermost.error.log;
    }

}

/config/nginx/proxy.conf:

client_max_body_size 10m;
client_body_buffer_size 128k;

#Timeout if the real server is dead
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;

# Advanced Proxy Config
send_timeout 5m;
proxy_read_timeout 240;
proxy_send_timeout 240;
proxy_connect_timeout 240;

# Basic Proxy Config
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Ssl on;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_redirect  http://  $scheme://;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_cache_bypass $cookie_session;
proxy_no_cache $cookie_session;
proxy_buffers 32 4k;

Error shown in mattermost log:

[2017/02/17 02:53:00 CET] [EROR] websocket connect err: websocket: origin not allowed
[2017/02/17 02:53:00 CET] [EROR] /api/v3/users/websocket:connect code=500 rid=rq54i7rnyprri8qgsod1kccaey uid=kmht6cdtttf67f4h8jifeiapjh ip=192.168.1.102 Failed to upgrade websocket connection [details: ]

Mattermost ver 3.6.2.

1 Like
2

Been running Mattermost for about a week now. Just started getting these messages too.

3

Also seeing this message in the desktop client:
Error during WebSocket handshake: Unexpected response code: 403

4

Same here. Configuration worked fine until i upgraded my gitlab omnibus from 8.16.3 to 8.17.0.

Anyone got a solution for this? Strangely, nginx logs show no errors.

5

The reason is most likely a new cross-origin check: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/1966
Fixed it by adding headers to the nginx forwarding:

 location /api/v3/users/websocket {
  proxy_pass            http://127.0.0.1:8065;
  (...)
  proxy_set_header      Host (your host name);
  proxy_set_header      X-Forwarded-For $remote_addr;
}
6

Thanks for posting your solution @sschlott! Appreciated :slight_smile:

@azb and @mhoffmeyer, does this solve your issue?

7

I’ve been trying to resolve this one for a while, currently using this package - https://github.com/mattermost/mattermost-docker
Have tried adding in the separate location for location /api/v3/users/websocket to no avail

8

IMO, the problem is that the WebSocket code does not honor the Mattermost CORS setting. Instead, it now applies a default check to see if the Origin header is the same as the Host header. See http://www.gorillatoolkit.org/pkg/websocket (scroll down to “Origin Considerations”)

9

Aren’t the headers already included in my proxy conf /config/nginx/proxy.conf correct?

proxy_set_header Host $host:$server_port;
...
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
10

I think @bradhowes is absolutely right. I had mm 3.6.2 running and had set
"AllowCorsFrom": "mydomain.tld",
and things were working fine.

After an upgrade to mm 4.0.3 I started getting websocket errors which seemed weird so after a couple of hours tinkering with the nginx reverse proxy I simply change above setting to:
"AllowCorsFrom": "*",
and everything works again.

11

Did you try “AllowCorsFrom”: “https://mydomain.tld”?

ie full URL including scheme passed in the origin header automatically by nginx.

12

I found that you need to include all the possible variations sent by clients:

13

and:

  • wss://mydomain.tld
  • wss://mydomain.tld:443

(new user limit of 2 links per post)