[SOLVED] Mattermost websocket error

Troubleshooting
#1

Hello,

Gitlab Mattermost is giving us the following error:

Please check connection, Mattermost unreachable. If issue persists, ask administrator to check WebSocket port.

And Gitlab logs show the following:

==> /var/log/gitlab/mattermost/mattermost.log <==
[2017/10/13 04:40:49 UTC] [EROR] /api/v4/websocket:connect code=500 rid=ycji7i9ht7838fk4hkpb5nbtdw uid=8z1d45uhotnuifu5pqb93gp5ca ip=10.96.2.86 Failed to upgrade websocket connection [details: ]
[2017/10/13 04:42:04 UTC] [EROR] websocket connect err: websocket: 'Origin' header value not allowed

==> /var/log/gitlab/mattermost/current <==
2017-10-13_04:42:04.62615 ip-10-96-15-151 mattermost: 2017/10/13 04:42:04 http: multiple response.WriteHeader calls
2017-10-13_04:42:04.62622 ip-10-96-15-151 mattermost: [2017/10/13 04:42:04 UTC] [EROR] websocket connect err: websocket: 'Origin' header value not allowed
2017-10-13_04:42:04.62625 ip-10-96-15-151 mattermost: [2017/10/13 04:42:04 UTC] [EROR] /api/v4/websocket:connect code=500 rid=dczoyarprjdnxreigi5tj4tr7a uid=8z1d45uhotnuifu5pqb93gp5ca ip=10.96.2.86 Failed to upgrade websocket connection [details: ]

It’s behind an ELB with the following configs and the SSL is terminating there (followed this advice).

ELB listeners are as follows:

17 AM

(I added the 8065 part just for testing).

Here’s my mattermost configs in gitlab.rb that’s relevant for this issue:

mattermost_external_url 'https://chat.company.com'
mattermost['enable'] = true
mattermost['service_address'] = "127.0.0.1"
mattermost['service_port'] = "8065"

mattermost_nginx['enable'] = true
mattermost_nginx['listen_port'] = 1234
mattermost_nginx['listen_https'] = false
mattermost_nginx['proxy_set_headers'] = {
  "Host" => "$http_host",
  "X-Real-IP" => "$remote_addr",
  "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
  "X-Frame-Options" => "SAMEORIGIN",
  "X-Forwarded-Proto" => "https",
  "X-Forwarded-Ssl" => "on",
  "Upgrade" => "$http_upgrade",
  "Connection" => "$connection_upgrade"
}

How can I get rid of this issue? What am I doing wrong here?

Thanks,

EDIT: We are on Gitlab Mattermost (version 4.2). Also, opened a ticket in Gitlab but they asked me to come and ask for help here.

#2

Hello @farahfa,

Thanks for your feedback,

Perhaps this websocket troubleshooting documentation will help?

You could also try updating to the latest Mattermost version which shipped today (10/16).

Let us know how it goes?

#3

Hi @lindy65,

Well, we are using Mattermost through the latest Gitlab Omnibus package. Also, Following that websocket troubleshooting page we can see that websocket’s status is 403.

websocket

We tried do a port forward (TCP) from 1234, which is the port that Mattermost is listening to (check the Gitlab nginx config above in the OP), to 8065 (Also TCP), but still the same error.

ELB

#4

Thanks for posting back your results @farahfa,

I’ll post this to the team and see if the devs have any suggestions for you.

#5

Hi @lindy65, anything new about this issue?

#6

@farahfa What is your mattermost['service_allow_cors_from'] config setting set to?

1 Like
#7

@farahfa Can you click on one of those connections in the developer console and confirm that the “Host” and “Origin” headers match each other?

11 PM

If not, you’ll need to add the origin to your “service_allow_cors_from” setting.

If they do match, double check any proxies you have in place to make sure that they’re forwarding those headers correctly. For example, if you’re using CloudFront, you need to ensure that it’s forwarding the Origin and Host headers.

1 Like
#8

Thanks, changing the mattermost['service_allow_cors_from'] fixed the issue for me.

1 Like