[Solved] Mattermost websocket error

Troubleshooting
1

Hello,

Gitlab Mattermost is giving us the following error:

Please check connection, Mattermost unreachable. If issue persists, ask administrator to check WebSocket port.

And Gitlab logs show the following:

==> /var/log/gitlab/mattermost/mattermost.log <==
[2017/10/13 04:40:49 UTC] [EROR] /api/v4/websocket:connect code=500 rid=ycji7i9ht7838fk4hkpb5nbtdw uid=8z1d45uhotnuifu5pqb93gp5ca ip=10.96.2.86 Failed to upgrade websocket connection [details: ]
[2017/10/13 04:42:04 UTC] [EROR] websocket connect err: websocket: 'Origin' header value not allowed

==> /var/log/gitlab/mattermost/current <==
2017-10-13_04:42:04.62615 ip-10-96-15-151 mattermost: 2017/10/13 04:42:04 http: multiple response.WriteHeader calls
2017-10-13_04:42:04.62622 ip-10-96-15-151 mattermost: [2017/10/13 04:42:04 UTC] [EROR] websocket connect err: websocket: 'Origin' header value not allowed
2017-10-13_04:42:04.62625 ip-10-96-15-151 mattermost: [2017/10/13 04:42:04 UTC] [EROR] /api/v4/websocket:connect code=500 rid=dczoyarprjdnxreigi5tj4tr7a uid=8z1d45uhotnuifu5pqb93gp5ca ip=10.96.2.86 Failed to upgrade websocket connection [details: ]

It’s behind an ELB with the following configs and the SSL is terminating there (followed this advice).

ELB listeners are as follows:

17 AM
17 AM2096×306 78.6 KB

(I added the 8065 part just for testing).

Here’s my mattermost configs in gitlab.rb that’s relevant for this issue:

mattermost_external_url 'https://chat.company.com'
mattermost['enable'] = true
mattermost['service_address'] = "127.0.0.1"
mattermost['service_port'] = "8065"

mattermost_nginx['enable'] = true
mattermost_nginx['listen_port'] = 1234
mattermost_nginx['listen_https'] = false
mattermost_nginx['proxy_set_headers'] = {
  "Host" => "$http_host",
  "X-Real-IP" => "$remote_addr",
  "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
  "X-Frame-Options" => "SAMEORIGIN",
  "X-Forwarded-Proto" => "https",
  "X-Forwarded-Ssl" => "on",
  "Upgrade" => "$http_upgrade",
  "Connection" => "$connection_upgrade"
}

How can I get rid of this issue? What am I doing wrong here?

Thanks,

EDIT: We are on Gitlab Mattermost (version 4.2). Also, opened a ticket in Gitlab but they asked me to come and ask for help here.

2

Hello @farahfa,

Thanks for your feedback,

Perhaps this websocket troubleshooting documentation will help?

You could also try updating to the latest Mattermost version which shipped today (10/16).

Let us know how it goes?

3

Hi @lindy65,

Well, we are using Mattermost through the latest Gitlab Omnibus package. Also, Following that websocket troubleshooting page we can see that websocket’s status is 403.

websocket

We tried do a port forward (TCP) from 1234, which is the port that Mattermost is listening to (check the Gitlab nginx config above in the OP), to 8065 (Also TCP), but still the same error.

ELB

4

Thanks for posting back your results @farahfa,

I’ll post this to the team and see if the devs have any suggestions for you.

5

Hi @lindy65, anything new about this issue?

6

@farahfa What is your mattermost['service_allow_cors_from'] config setting set to?

7

@farahfa Can you click on one of those connections in the developer console and confirm that the “Host” and “Origin” headers match each other?

11 PM

If not, you’ll need to add the origin to your “service_allow_cors_from” setting.

If they do match, double check any proxies you have in place to make sure that they’re forwarding those headers correctly. For example, if you’re using CloudFront, you need to ensure that it’s forwarding the Origin and Host headers.

8

Thanks, changing the mattermost['service_allow_cors_from'] fixed the issue for me.

9

I get this error. Any tips to further troubleshoot this within the MM desktop app?

image
image966×342 13.6 KB

10

Hi @ashley-mort and welcome to the Mattermost forums!

This topic is already solved, please create a new one and describe the problem you’re seeing with a few more details, like the server version you’re using, the desktop app version and what exactly the problem is. In your screenshot f.ex. I cannot see an error message.

11

@farahfa

Where do you check this mattermost[‘service_allow_cors_from’]

I couldn’t find this line in the config file