[Solved] Error: Please check connection, Mattermost unreachable. If issue persists, ask administrator to check WebSocket port

[torbengb]

Mod Note:
For a more broad walkthrough troubleshooting general Mattermost connection issues, refer to this thread

Troubleshooting Mattermost Server Connectivity Issues

Solving Connectivity Problems with Your Mattermost Server - A Comprehensive Guide Having connection issues with your Mattermost server? Start here! Verify Server Status Start by checking the status of your Mattermost server. If the server is down, it would be unable to accept new connections. Use the ping command to verify server status. Learn More Check Mattermost Server Logs The server logs can provide valuable insight into what might be causing connection issues. Make s…

---

Summary
Fresh install of Mattermost team edition v7.8 hosted on Docker, getting warning banner:

Please check connection, Mattermost unreachable. If issue persists, ask administrator to check WebSocket port.

Steps to reproduce

• (internet) > (gateway) > (reverse proxy) > (server) > (docker) > (mattermost)
• (gateway) only allows 80+443; this is directed to a reverse proxy. (I have now added ports 8443 and 8065 to point straight to my server.)
• (reverse proxy) handles SSL, and it directs “https … chat.mydomain.tld” to server “http … 192.xxxx:8065”
• (server) is a Linux server running Docker with several containers. Mattermost is just one of many.
• Mattermost deployed using steps 1…4 given here: mattermost-com/deploy/
• docker-compose.yml edited to include values from .env (see file here)
• .env file edited to match my installation. (see file here)

Expected behavior
No warning banner.

Observed behavior
Log contains many lines like this:

DBG DBG web/context.go:113 > timestamp=2023-02-21 11:58:11.189 Z msg=Failed to upgrade websocket connection. path=/api/v4/websocket request_id=78z8jntyabdm5p5kdiw9b5td5w ip_addr=192.168.1.11 user_id=r8m9gk6mgfb7te3i79pco6r59h method=GET err_where=connect http_code=400 error=connect: Failed to upgrade websocket connection., websocket: the client is not using the websocket protocol: ‘upgrade’ token not found in ‘Connection’ header

[jgooge]

I’m having this same issue when using the latest version of the Linux desktop client. I can access the mattermost server via the web with no issues.

[torbengb]

That sounds … strange. This is clearly a server-side issue, so how can one client work OK and another does not? On my installation, I see the same error both in web and in desktop clients.

Surely there is some server-side stuff needed, but I don’t see what. Port-forwarding? Docker port configuration? Something else? I do not know the first thing about “websockets”.

[shurrman (Alexandr A. Alexandrov)]

The error is quite clear - your reverdse proxy does not handle websocket connections.
I have a similar setup, and using NGinx as a reverse proxy. Note my settings for the MTM location:

    location / {
        proxy_pass https://127.0.0.1:8443;
#        proxy_pass http://127.0.0.1:8088;
        proxy_set_header Host $http_host;
        proxy_set_header CLIENT_HOST $remote_addr;
        proxy_set_header Origin "";
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_ssl_verify              off;
    }

[torbengb]

Thank you for the input, shurrman! I don’t see that the error is that clear, it does not mention any port?
“Failed to upgrade websocket connection”
“‘upgrade’ token not found in ‘Connection’ header”
How does this tell me that it’s about the reverse proxy?
More importantly, how do I fix this? As stated, I have already port-forwarded 8065 and 8443 directly from my gateway to my server. Is that not it? What more is needed?

For reference, the reverse proxy I currently use is the “Application Portal” built into my Synology NAS. Is “websocket” something special, something different than HTTP(S) requests, that this reverse proxy application does not support?

[shurrman (Alexandr A. Alexandrov)]

You have this in your error message:

error=connect: Failed to upgrade websocket connection., websocket: the client is not using the websocket protocol: ‘upgrade’ token not found in ‘Connection’ header

And I posted my reverse-proxy config, that handles exactly websockets-relarted stuff:

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

See - I set appropriate headers. I know nothing about what proxy is under the hood in your NAS and is/how it configurable.
Websocket is not http, but it “starts” over http.

[shurrman (Alexandr A. Alexandrov)]

If your reverse proxy can not handle Websockets (which I assume is true), probably the only thing you could try is to point your Gateway → Server, and run docker-compose version of MTM with Nginx

docker-compose -f docker-compose.yml -f docker-compose.nginx.yml up -d

[torbengb]

@shurrman

probably the only thing you could try is to point your Gateway → Server

I think your suggestion will work to prove the Mattermost installation, but that method is impractical for actual use because no other services would be available; without any reverse proxy to direct traffic, I would then be running only that service.

I am looking to replace the Synology reverse proxy with Traefik instead, but that’s another separate project and I do not know whether Traefik supports websockets!

I am clearly not a network engineer, so this is frustratingly complicated. Thank you for your help!

[agriesser (Alexander Griesser)]

Hi everyone and welcome to the Mattermost forums

The desktop application is connecting through the same network path than the web app, technically, right?

Client → Browser → Synology → Mattermost
Client → Desktop App → Synology → Mattermost

Or do you use different ways/URLs to connect then?
If your client is able to establish the websocket connection using the browser, it should also be able to establish it using the desktop app.

[torbengb]

@agriesser

The desktop application is connecting through the same network path than the web app, right?

Yes, I should think so. I just point the browser to the URL (https://chat.mydomain.tld/myorganisation), and I use the same information in the desktop/android app.

[agriesser (Alexander Griesser)]

Hmm… the /myorganisation at the end confuses me. According to your .env file, there is no /myorganisation part at the end.

Your SiteUrl is set to https://chat.g~~~~o.e~ without the /myorganisation at the end, the SSL certificate also looks fine, so I’m unsure about what the problem with the websockets could be.
When you open the menu of the desktop app, there’s an option to view the developer tools for the current server:

Would be interesting to see if there are any error messages in the console.

[torbengb]

The “/myorganisation” is the team name that I must enter at the very start of setting up Mattermost. In my case it’s actually “/gb” and you can see it’s there:

I found the developer tools (looks like Firefox) and it has hundreds of entries like this:

and sometimes (but rarely) also entries like this:

I’m not familiar with this. Is there anything particular that you’d like me to look at?

[agriesser (Alexander Griesser)]

You do not need to add the team name to the URL for connecting in the desktop app, maybe this is also the problem here. Please just use https://chat.yourdomain.eu, the desktop app will automatically add the teamname then once you logged in.

Can you remove the profile in your desktop app and try to create a new one with just the domain name in the connection details and let me know if that changes anything with regrads to the websockets?
If it doesn’t: Can you expand the error messages you’re seeing in the console here? Are there any more details available as to why the connect has been closed f.ex.?

[torbengb]

First, I removed the existing server connection. I was sent to the normal welcome screen. This is expected. I entered the server URL, and I did not enter the “/team” part of the URL at the end. After some seconds, the familiar red error banner appears. See screenshots here (Imgur).

Expanding the error message does not reveal a lot. Is this useful?

[torbengb]

I SOLVED IT! Synology DSM has a setting to create websockets. I did that, refreshed the web app, and now the ugly red banner is gone!!

[agriesser (Alexander Griesser)]

Thanks, that’s valuable information for others that might have this same problem!

[charlie (Charles Chiu)]

Thanks! This setting solved my websocket warning in http mode too.

[shurrman (Alexandr A. Alexandrov)]

Great to know (and I think my hint was the clue…)
But having Synology as a proxy/router… might be not the best idea… However it is solely your choice/way, and again glad to know you solved it

[shurrman (Alexandr A. Alexandrov)]

MTM uses WebSockets, WebSockets use/need headers with that “upgrade” stuff

[torbengb]

Synology as a proxy might be not the best idea

You are absolutely correct, and that is yet another work in progress. (So many ongoing projects in my home lab!) Using built-in functionality of Synology was an easy way to get started.

I am working on transitioning to Traefik running on a separate server machine, but of course Traefik is also a whole can of worms. Or, instead of Traefik, use Nginx directly, which is even more complicated. After all, people can make a full-time living based on these things; no wonder it’s a bit overwhelming for an average IT guy!

I will get there eventually. This was one important step.