Websocket issues with Mattermost 6.0.0 and Boards: Mattermost Websocket client is not ready

Troubleshooting
1

Summary
Websocket issues with Mattermost 6.0.0: WSClient error on open: Mattermost Websocket client is not ready
Some UI actions (e.g. adding a new board in Boards) are not working and require a page refresh (or changing channel and then coming back to the original one) to fetch updated data, maybe because the underlying websocket connection is not working as intended?

Steps to reproduce
Deploy mattermost 6.0.0 on a microk8s cluster using mattermost operator, standard ingress-nginx and a custom ingress with a wildcard SSL cert provisioned by cert-manager.

Expected behavior
Websocket connection should work.

Observed behavior
After upgrading to Mattermost 6.0.0, I noticed that some websocket errors came up when accessing Boards. More specifically, as soon as I access that I get errors in Chrome’s console
After some time, Boards shows an error as well.

However, it looks like the websocket connection is working as intended when I am on a regular channel (e.g. outside of Boards)

Please see all 3 screenshots here: Mattermost — ImgBB
Being a new user, I cannot post more than 1 pic and 2 links so I grouped them together.

Did anyone encounter similar issues? My custom ingress configuration is pretty standard

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mm-tls
  namespace: mattermost
  annotations:
    # nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/proxy-body-size: 1000M
    # cert-manager.io/cluster-issuer: "letsencrypt"
spec:
  ingressClassName: public
  tls:
  - hosts:
    - <host>
    secretName: <secret name>
  rules:
  - host:<host>
    http:
      paths:
      - path: "/"
        pathType: Prefix
        backend:
          service:
            name: mm-app
            port:
              number: 8065

Thanks!

2

Hi Luca!

Thanks for the detailed report! Can you please check if you have the ServiceSettings.EnableReliableWebSockets server configuration property set to true?

Reliable websockets needs to be enabled for Boards to run properly on v6

3

Hey, thanks for the quick answer. Unfortunately, I have them enabled

"EnableReliableWebSockets": true.

If it can help, I upgraded from 5.37 to 6.0 through 5.38 and 5.39 this morning.

4

Thanks for the info! Can you please post a full log from the browser after refreshing on a board page and another screenshot similar to the one you posted earlier of the websocket messages on this same refresh?

5

Sure! Here are the logs:

index.js:114 Loading plugin focalboard, version 0.9.2
index.js:114 Loading plugin com.mattermost.walltime-plugin, version 0.1.1
index.js:114 Loading plugin standup-raven, version 3.2.2
index.js:114 Loading plugin com.mattermost.nps, version 1.1.0
index.js:114 Loading plugin com.mattermost.plugin-channel-export, version 1.0.0
index.js:114 Loading plugin playbooks, version 1.20.1
index.js:100 Loaded plugin com.mattermost.plugin-channel-export, version 1.0.0
index.js:100 Loaded plugin com.mattermost.nps, version 1.1.0
index.js:100 Loaded plugin com.mattermost.walltime-plugin, version 0.1.1
focalboard_422f9e939fe02632_bundle.js:2 [1634130400.97] OctoClient baseURL: https://mattermost.sungazer.io/plugins/focalboard
index.js:100 Loaded plugin focalboard, version 0.9.2
index.js:100 Loaded plugin playbooks, version 1.20.1
react-dom.development.js:24994 Download the React DevTools for a better development experience: https://fb.me/react-devtools
index.js:100 Loaded plugin standup-raven, version 3.2.2
instrument.js:110 [1634130401.23] WSClient initialised for plugin id "focalboard"
instrument.js:110 websocket connecting to wss://mattermost.sungazer.io/api/v4/websocket
instrument.js:110 [1634130401.24] WSClient Mattermost Websocket not ready, retrying
instrument.js:110 [1634130401.24] WSClient initialised for plugin id "focalboard"
instrument.js:110 [1634130401.24] WSClient initialised for plugin id "focalboard"
instrument.js:110 [1634130401.24] WSClient initialised for plugin id "focalboard"
instrument.js:110 [1634130401.31] attachToBoard: cc00df92-fb85-4617-83d9-fa4331593620
instrument.js:110 [1634130401.31] attachToBoard: cc00df92-fb85-4617-83d9-fa4331593620
instrument.js:110 [1634130401.39] Manual sort
instrument.js:110 [1634130401.41] OctoClient baseURL: https://mattermost.sungazer.io/plugins/focalboard
instrument.js:110 [1634130401.41] attachToBoard: cc00df92-fb85-4617-83d9-fa4331593620
instrument.js:110 [1634130401.54] WSClient Mattermost Websocket not ready, retrying
instrument.js:110 [1634130401.84] WSClient Mattermost Websocket not ready, retrying
instrument.js:110 [1634130402.14] WSClient Mattermost Websocket not ready, retrying
instrument.js:110 [1634130402.44] WSClient Mattermost Websocket not ready, retrying
instrument.js:110 [1634130402.74] WSClient Mattermost Websocket not ready, retrying
instrument.js:110 [1634130403.04] WSClient Mattermost Websocket not ready, retrying
instrument.js:110 [1634130403.35] WSClient Mattermost Websocket not ready, retrying
instrument.js:110 [1634130403.64] WSClient Mattermost Websocket not ready, retrying
instrument.js:110 [1634130403.95] WSClient Mattermost Websocket not ready, retrying
instrument.js:110 [1634130404.25] WSClient error on open: Mattermost Websocket client is not ready
(anonymous) @ instrument.js:110

And the screenshot:

image
image479Ă—579 14.8 KB

6

Please also have a look at this video that shows the Boards issues that may be linked to this websocket issue: Mattermost Boards Issue - YouTube (whenever the page flashes white, I was refreshing it manually)

7

Thanks again for the detailed answer!

From the frontend logs I’m missing a got connection id X message that gets logged when the websocket connection is initially established. Just to rule the reliable websockets out in case the server is not picking up the configuration, could you please run:

mmctl --local config get ServiceSettings.EnableReliableWebSockets

from one of the nodes of the cluster to see if it comes out true?

Besides that, what’s the browser/client that you’re using to access boards? how are you getting to the board? (from the icon top right of a channel, from the application header, changing to boards, etc)

8

Huh, this is weird. After enabling local mode, I ran the mmctl command but the setting was false, even though it was true in the config.json!

~ $ mmctl --local config get ServiceSettings.EnableReliableWebSockets
false
~ $ cat config/config.json | grep -i EnableReliable
    "EnableReliableWebSockets": true
~ $ printenv | grep MM_
MM_METRICSSETTINGS_ENABLE=true
MM_FILESETTINGS_DRIVERNAME=amazons3
MM_FILESETTINGS_AMAZONS3SSL=true
MM_SERVICESETTINGS_SITEURL=<redacted>
MM_FILESETTINGS_MAXFILESIZE=1048576000
MM_CONFIG=<redacted>
MM_LOGSETTINGS_FILELEVEL=INFO
MM_CLUSTERSETTINGS_ENABLE=true
MM_METRICSSETTINGS_LISTENADDRESS=:8067
MM_FILESETTINGS_AMAZONS3BUCKET=mattermost2
MM_INSTALL_TYPE=kubernetes-operator
MM_LOGSETTINGS_CONSOLELEVEL=INFO
MM_FILESETTINGS_AMAZONS3ENDPOINT=<redacted>
MM_FILESETTINGS_AMAZONS3ACCESSKEYID=<redacted>
MM_FILESETTINGS_AMAZONS3SECRETACCESSKEY=<redacted>
MM_SERVICESETTINGS_ENABLELOCALMODE=true
MM_CLUSTERSETTINGS_CLUSTERNAME=production
MM_PLUGINSETTINGS_ENABLEUPLOADS=true

Anyway, I set the reliablewebsockets option from env now and it looks like it fixed things, as now the Boards UI reacts when I e.g. create a Board, and I also saw the “got connection id X” message in the console. Thanks for pointing me to the mmctl command, I didn’t think about looking there and I only checked the config.json!

9

So, after enabling local mode I ran the command, and I found something weird:

~ $ mmctl --local config get ServiceSettings.EnableReliableWebSockets
false
~ $ cat config/config.json | grep -i EnableReliable
    "EnableReliableWebSockets": true
~ $ printenv | grep MM_
MM_METRICSSETTINGS_ENABLE=true
MM_FILESETTINGS_DRIVERNAME=amazons3
MM_FILESETTINGS_AMAZONS3SSL=true
MM_SERVICESETTINGS_SITEURL=<redacted>
MM_FILESETTINGS_MAXFILESIZE=1048576000
MM_CONFIG=<redacted>
MM_LOGSETTINGS_FILELEVEL=INFO
MM_CLUSTERSETTINGS_ENABLE=true
MM_METRICSSETTINGS_LISTENADDRESS=:8067
MM_FILESETTINGS_AMAZONS3BUCKET=mattermost2
MM_INSTALL_TYPE=kubernetes-operator
MM_LOGSETTINGS_CONSOLELEVEL=INFO
MM_FILESETTINGS_AMAZONS3ENDPOINT=<redacted>
MM_FILESETTINGS_AMAZONS3ACCESSKEYID=<redacted>
MM_FILESETTINGS_AMAZONS3SECRETACCESSKEY=<redacted>
MM_SERVICESETTINGS_ENABLELOCALMODE=true
MM_CLUSTERSETTINGS_CLUSTERNAME=production
MM_PLUGINSETTINGS_ENABLEUPLOADS=true

Basically, mmctl is telling me that the setting is false, but it is true in config.json.
I now set the settings through the env var MM_SERVICESETTINGS_ENABLERELIABLEWEBSOCKETS and it looks like it fixed things (UI responds to changes and I saw the got connection id X in the logs).

Thanks a lot for pointing me to mmctl, I didn’t think about it and I only looked at the config.json!

Let me know if I can be of further help for troubleshooting this.

Thanks again!

1 Like
10

Glad to know it’s working now!!

Cluster setups tipicaly use the configuration in database feature to share the configuration amongst all the nodes, so that might be a reason why you’re seeing different values in the server config and in the config.json file, which would not be the source of truth anymore. Setting the value through env variables is a perfect long term solution, and you could use mmctl --local config set as well in case you need to modify the configuration again :slight_smile:

11

Just came here because I had the same problem but in my case the solution was to enable reliable websockets in config.json.
I would suggest to mention this in the upgrade docs, because it seems like an important information.

12

This change did work for me:

1 - Set to “true” the EnableReliableWebSockets env option in {install_path}/config/config.json
2 - Restart mattermost service

1 Like
13

Where do you set the MM_ env variables?