Upgraded from 3.7 to 3.8 broke websockets! help!

ChrisFernandez · April 20, 2017, 12:17am

Hello everything was working fine… I upgraded like I always do… log in… got a msg to add a URL in the console/configuration field… I added my url… and after that I get:
websocket connect err: websocket: origin not allowed
/api/v3/users/websocket:connect code=500 rid=4che7w3383r88qxswssi53nkwh uid= ip=107.182.x.x Failed to upgrade websocket connection [details: ]

I changed the URL to all my interfaces and restarted with same issue…

MY setup?

nginx port 443 port 80 redirect to 443
config for nginx:

server {
      listen 443 ssl;
      server_name chat.mydomain.info o5imyonion.onion chat.mydomain;

      ssl on;
      ssl_certificate  /etc/letsencrypt/live/chat.mydomain.info/fullchain.pem;
      ssl_certificate_key /etc/letsencrypt/live/chat.mydomain.info/privkey.pem;
      ssl_dhparam /home/mattermost/cert/dhparam.pem;
      ssl_session_timeout 5m;
      ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
      ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
      ssl_prefer_server_ciphers on;
      ssl_session_cache shared:SSL:10m;


      location / {
        gzip off;
        proxy_set_header X-Forwarded-Ssl on;
        client_max_body_size 50M;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Frame-Options SAMEORIGIN;
        proxy_pass http://127.0.0.1:8065;
     }
   }

on mattermost I have added this two changes…
on the URL “enforced” field now…(I dont know why you guys force this… when it SHOULD be an option to admins… not you…
I have http://127.0.0.1 (had my external ip, my external domains all of them, this is just the last one I tried)
for port I have:
127.0.0.1:8065

why? nginx is forwarding the connection to localhost:8065 as you can see in the nginx config above…

Please help all my users are complaining …

JeffSchering · April 20, 2017, 10:11pm

I notice that your NGINX config doesn’t have a location for /api/v3/users/websocket… maybe that’s the issue?

The install docs say to have the following config for /api/v3/users/websocket:

location /api/v3/users/websocket {
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       client_max_body_size 50M;
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       proxy_read_timeout 600s;
       proxy_pass http://backend;
   }

ChrisFernandez · April 20, 2017, 10:52pm

Interesting, I got my config from one of the first releases so this maybe have changed so I must to miss it, in any case thanks for noticing, I will test tonight and report back in case someone have same issue.
BTW do I need to add that or substitute what I have with “/”?

Thanks

JeffSchering · April 20, 2017, 11:24pm

You add it. The instructions are here: https://docs.mattermost.com/install/install-ubuntu-1604.html#configuring-nginx-as-a-proxy-for-mattermost-server. Although they are in the Ubuntu instructions, the config is the same for every OS. Also, the TLS config might be of interest: https://docs.mattermost.com/install/install-ubuntu-1604.html#configuring-nginx-with-ssl-and-http-2

I’m very interested to see what the solution turns out to be so that I can update the documentation.

mouson · April 21, 2017, 6:50am

I have same problem,

there is my mattermost.log

[2017/04/21 14:46:36 CST] [EROR] websocket connect err: websocket: origin not allowed
[2017/04/21 14:46:36 CST] [EROR] /api/v3/users/websocket:connect code=500 rid=gj88f7qtu3yd3m5kdebw11ot9w uid=rhynu9jrebr1zcgpo4t1gwag8w ip=172.16.2.254 Failed to upgrade websocket connection [details: ]

congalong · April 21, 2017, 9:06pm

https://github.com/mattermost/platform/issues/6183

I tried building it locally but not sure if I built it wrong or what, but it didn’t seem to fix my issue.

congalong · April 21, 2017, 10:09pm

Just tested 3.8.2. Still seeing the same issue.

I have a nginx reverse proxy. Was working on 3.7.3

[2017/04/21 15:08:58 PDT] [EROR] websocket connect err: websocket: origin not allowed
[2017/04/21 15:09:44 PDT] [EROR] /api/v3/users/websocket:connect code=500

My nginx config

upstream mattermost {
  server 127.0.0.1:8065;
}

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;

server {
  listen *:80;
  server_name chat.mycomp.com;
  server_tokens off;
  return 301 https://chat.mycomp.com:443$request_uri;
  access_log  /var/log/gitlab/nginx/mattermost_access.log;
  error_log   /var/log/gitlab/nginx/mattermost_error.log;
}

server {
  listen *:443 ssl http2;
  server_name chat.mycomp.com;
  server_tokens off;     # don't show the version number, a security best practice
  client_max_body_size 250m;

  ssl on;
  ssl_certificate /etc/gitlab/ssl/mycomp-chained.crt;
  ssl_certificate_key /etc/gitlab/ssl/mycomp.key;
  ssl_session_timeout 5m;
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:10m;


  ## Real IP Module Config
  ## http://nginx.org/en/docs/http/ngx_http_realip_module.html

  access_log  /var/log/gitlab/nginx/mattermost_access.log;
  error_log   /var/log/gitlab/nginx/mattermost_error.log;

  location /api/v3/users/websocket {
 proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       client_max_body_size 50M;
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       proxy_read_timeout 600s;
       proxy_pass http://mattermost;
   }

   location / {
       client_max_body_size 50M;
       proxy_set_header Connection "";
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       proxy_read_timeout 600s;
       proxy_cache mattermost_cache;
       proxy_cache_revalidate on;
       proxy_cache_min_uses 2;
       proxy_cache_use_stale timeout;
       proxy_cache_lock on;
       proxy_pass http://mattermost;
   }

}

jasonblais · April 21, 2017, 11:10pm

@ChrisFernandez and @congalong: Mattermost 3.8.2 is available for download here: http://about.mattermost.com/download/

The dot release should fix the WebSocket connection issues you’ve described.

Please let us know if you hit any further issues, and thank you for the report!

PS: @congalong You may have grabbed an invalid build for 3.8.2: the first release build automation didn’t finish successfully, so we re-launched it afterwards. Can you try again, taking the build from http://about.mattermost.com/download/?

mouson · April 22, 2017, 7:25am

Thanks for your help.

I update our Mattermost to 3.8.2 and use browser client
but got same error message.

[2017/04/22 15:20:32 CST] [EROR] websocket connect err: websocket: origin not allowed
[2017/04/22 15:20:32 CST] [EROR] /api/v3/users/websocket:connect code=500 rid=88ybn74wqib9mj5c6joixicjze uid=k84pboggc7f5fd944ry35cugky ip=36.225.139.27 Failed to upgrade websocket connection [details: ]

yoshtec · April 22, 2017, 2:54pm

Same here (using community edition 3.8.2)

yoshtec · April 23, 2017, 6:38pm

BTW does the Config for nginx have to be changed to also accommodate the new /api/v4/… location?

jasonblais · April 24, 2017, 4:54pm

@mouson @yoshtec

Do you have AllowCorsFrom setting set for cross-origin requests: https://docs.mattermost.com/administration/config-settings.html#enable-cross-origin-requests-from

If you’re using multiple domains, you should add all the alternative domains to AllowCorsFrom setting after which the websockets would work. Otherwise you’ll get websocket connect err: websocket: origin not allowed errors.

Let us know if the issue keeps persisting

mouson · April 24, 2017, 6:25pm

@jasonblais

Thank you for your help.

When i type "AllowCorsFrom": "*", websockets is work,
but i type "AllowCorsFrom": "*.mydomain", "AllowCorsFrom": "same site url domain" or "AllowCorsFrom": "" websockets does not work.

jasonblais · April 27, 2017, 1:53pm

@mouson what’s your Site URL set to?

mouson · April 27, 2017, 4:11pm

Our config.json like:

"SiteURL": "https://our.domain.com"
"AllowCorsFrom": "*",

that is work

"SiteURL": "https://our.domain.com"
"AllowCorsFrom": "*.domain.com"

or

"SiteURL": "https://our.domain.com"
"AllowCorsFrom": "our.domain.com"

that is not work

jasonblais · April 27, 2017, 4:24pm

Thanks @mouson, can you try setting “AllowCorsFrom”: “https://our.domain.com” ?

mouson · April 27, 2017, 5:14pm

Thanks @jasonblais,
I am try to set "AllowCorsFrom": "https://our.domain.com"
websocket not work

jasonblais · April 27, 2017, 6:56pm

Hmm, okay…that should have worked

Which OS did you use to install Mattermost? Is it running on Ubuntu?
Which DB are you running Mattermost on (MySQL, Postgres, or something else)?
Are these the error messages you’re seeing in the logs? (you posted them earlier, just checking to see if they have changed)

[2017/04/22 15:20:32 CST] [EROR] websocket connect err: websocket: origin not allowed
[2017/04/22 15:20:32 CST] [EROR] /api/v3/users/websocket:connect code=500 rid=88ybn74wqib9mj5c6joixicjze uid=k84pboggc7f5fd944ry35cugky ip=36.225.139.27 Failed to upgrade websocket connection [details: ]

Can you confirm the Mattermost server version number from Mattermost Main Menu > About Mattermost dialog?

Just want to confirm the upgrade to 3.8.2 was successful.

mouson · April 28, 2017, 3:55am

Thank for @jasonblais help.

We have two Server, 1. Mattermost service, 2. Nginx Reverse Proxy Service

Mattermost Service Service:

ENV:

OS: CentOS release 6.9 - 64bit
DB: postgresql-9.5
Local IP: 172.16.7.22
Mattermost

[mattermost]# ./bin/platform version
Version: 3.8.0
Build Number: 3.8.2
Build Date: Fri Apr 21 21:30:35 UTC 2017
Build Hash: dd1db760e378d2df7a52976e812f77f54f85c2c0
Build Enterprise Ready: true
DB Version: 3.8.0

error log on mattermost is same the previous

Nginx Reverse Proxy

OS: CentOS release 6.8 (Final) / 64bit
WEB SERVER: nginx/1.11.12
local ip: 172.16.7.20
nginx config:

upstream mattermost {
   server 172.16.7.22:8065;
}
server {
    listen 80;
    server_name    ourdomain.com;
    return 301 https://$server_name$request_uri;
}
server {
    listen 443 ssl http2;
    server_name ourdomain.com;

    ssl on;
    ssl_certificate /etc/letsencrypt/live/ourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ourdomain.com/privkey.pem;
   ssl_session_timeout 5m;
   ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
   ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
   ssl_prefer_server_ciphers on;
   ssl_session_cache shared:SSL:10m;

   access_log  /var/log/nginx/ourdomain_access.log;
   error_log   /var/log/nginx/ourdomain_error.log;

   location /api/v3/users/websocket {
       access_log  /var/log/nginx/ourdomain_websocket_access.log;
       error_log  /var/log/nginx/ourdomain_websocket_error.log;

       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       client_max_body_size 50M;
       proxy_set_header Origin $remote_addr;
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       proxy_read_timeout 600s;
       proxy_pass http://mattermost;
   }

   location / {
       client_max_body_size 50M;
       proxy_set_header Connection "";
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       proxy_read_timeout 600s;
       proxy_cache mattermost_cache;
       proxy_cache_revalidate on;
       proxy_cache_min_uses 2;
       proxy_cache_use_stale timeout;
       proxy_cache_lock on;
       proxy_pass http://mattermost;
   }
}

jasonblais · May 1, 2017, 12:23pm

@mouson

Thanks!

Just to confirm - are you using only one domain for your Mattermost instance?
Can you try "AllowCorsFrom": "", which disables cross-origin requests?
Can you also try "AllowCorsFrom": "domain.com", which should work (I touched base with one of the core developers)

Topic		Replies	Views
How to fix websocket error? Troubleshooting	12	22535	August 20, 2018
[SOLVED] The famous "Failed to upgrade websocket connection" Installation	4	12564	May 29, 2017
Failed to upgrade websocket connection: websocket: request origin not allowed by Upgrader.CheckOrigin Troubleshooting	2	1603	July 29, 2024
Websocket problem Docs - Help Wanted	1	1825	July 22, 2021
[SOLVED] Websockets not connecting because insecure Websocket endpoint Troubleshooting	4	23168	December 5, 2016

Upgraded from 3.7 to 3.8 broke websockets! help!

Related topics