Helm Chart on RKE2 / Ingress configuration for Websocket

Troubleshooting
1

Summary
Websocket connexion problem, is the problem coming from the ingress configuration?

Steps to reproduce
Install RKE2 v2.7.4 (K8S v1.23.17) + Charts: mattermost-team-edition (6.6.49)

Expected behavior
Have websocket working

Observed behavior
Please check connection, Mattermost unreacheable. If issue persists, ask administrator to check WebSocket port.

image
image1275×200 11.7 KB

As many of you, we have a problem with the websockets.
Probably from nginx/ingress configuration.

Here is the configuration:

ingress:
  annotations:
    cert-manager.io/cluster-issuer: keycloak-letsencrypt-prod
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/server-snippets: |
      location ~ /api/v[0-9]+/(users/)?websocket$ {
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       client_max_body_size 50M;
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       client_body_timeout 60;
       send_timeout 300;
       lingering_timeout 5;
       proxy_connect_timeout 90;
       proxy_send_timeout 300;
       proxy_read_timeout 90s;
       proxy_http_version 1.1;
       proxy_pass http://mattermost-team-edition.mattermost.svc.cluster.local:8065;
      }

      location / {
       client_max_body_size 50M;
       proxy_set_header Connection "";
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       proxy_read_timeout 600s;
       proxy_http_version 1.1;
       proxy_pass http://mattermost-team-edition.mattermost.svc.cluster.local:8065;
      }
  className: ''
  enabled: true
  hosts:
    - mattermost.domain.net
  path: /
  tls:
    - hosts:
        - mattermost.domain.net
      secretName: mattermost.domain.net-tls

We have added those from the nginx configuration here: Configure NGINX as a proxy for Mattermost server — Mattermost documentation

Anyone has a working helm chart values to suggest to have our Ingress OK with websockets?

We have already spent a lot of time to make this work.
Any suggestion would be highly appreciated :slight_smile:

2

Mattermost 6.6.49? Are you sure? There was a 6.6.2 but that hasn’t been supported for two years. I’d strongly recommend getting a newer version from Version Archive — Mattermost documentation

One thing that might be involved is, last year we began enforcing a security setting around cross-origin requests. Integrations configuration settings — Mattermost documentation

Did WebSockets ever work? It’s possible that network infrastructure or firewalls allow HTTP/HTTPS but not WS/WSS

3

@john.oliver

The helm chart version is mattermost-team-edition:6.6.49
The Mattermost version is 9.5.2.

Did WebSockets ever work? not yet here… I’m quite interested to have an example of helm values that makes it work using the proper nginx configuration.

Thanks!

4

Are we certain there’s no RKE load balancer involved? Or any external network issue? Something between your RKE environment and the Internet?

5

Nothing… Maybe a firewall but TCP 80 and 443 are open.

6

Anyone has managed to make the helm chart work?
How can WS work if the default ingress configuration do not have the upgrade feature for WSS?
Information is here:

And also in the Nginx configuration from the Mattermort website here:
https://docs.mattermost.com/install/config-proxy-nginx.html

7

Not sure what you mean by this. You may want to get HTTP / WS working, then add in the certs and enable TLS. Sorry, I’m not a k8s expert… this probably is an nginx issue, maybe https://forum.nginx.org/ can help?

8

After investigation, WSS can reach the app.

But I get this error message:
{“timestamp”:“2024-04-08 06:08:22.451 Z”,“level”:“debug”,“msg”:“Failed to upgrade websocket connection.”,“caller”:“web/context.go:111”,“path”:“/api/v4/websocket”,“request_id”:“”,“ip_addr”:“”,“user_id”:",“method”:“GET”,“err_where”:“connect”,“http_code”:400,“error”:“connect: Failed to upgrade websocket connection., websocket: request origin not allowed by Upgrader.CheckOrigin”}

Continuing to see how to properly configure nginx in order to prevent “request origin not allowed by Upgrader.CheckOrigin”

9

https://docs.mattermost.com/configure/integrations-configuration-settings.html#integrate-allowcorsfrom

10

Thank you…
Still trying to figure out how to put something different than *.
Highly appreciated.

11

Start with “*” and see if that resolves the issue.

Assuming it does, I asked and there was some confusing about comma-delineated vs. space delineated. Try:

http.cors.allow-origin: "http://localhost,http://127.0.0.1"

Or:

http.cors.allow-origin: "http://localhost http://127.0.0.1

And please LMK which or if both worked!

12

None of these worked in K8S. How can I see origin in the logs of the application?

13

Ctrl-Shift-J should open a Javascript console and show a lot of details. I’m not a developer and don’t “get” most of the output.

14

I mean from server side :-).
I will investigate and share the information when/if I find something.

Regards,

Francois