Websocket Error on Kubernetes

Troubleshooting
1

Hello Everybody,

i know this is another topic on the websocket error, but i have not found a solution to my problem on this forum or the web.
I am having the websocket error, which was often discussed on this forum. In my case, this is happening on an kubernetes cluster, where i installed mattermost via the official helm chart.

The error message I am getting is:

{"timestamp":"2022-03-31 21:10:41.886 Z","level":"error","msg":"Failed to upgrade websocket connection.","caller":"web/context.go:105","path":"/api/v4/websocket","request_id":"a1us537gp3b7xxyoawiuksyrmr","ip_addr":"xxxxx","user_id":"xxxxxx","method":"GET","err_where":"connect","http_code":500,"err_details":"websocket: the client is not using the websocket protocol: 'upgrade' token not found in 'Connection' header"}

I already posted this issue on the github repo of the helm chart, but i am not getting any respone.
I think this has something todo with the ingress, as the same configuration worked on an older kubernetes cluster with the v1beta1 ingress definition.

Any ideas where the problem is coming from or how to resolve it?
Thanks for any advice

Kubernetes version: v1.23.1
Ingress: Nginx
Ingress version: v1
LB: metallb
Cluster Type: Bare Metal

2

The nginx proxy is not upgrading the connection properly.

Can you post your nginx config here?

It should have this block:

location ~ /api/v[0-9]+/(users/)?websocket$ {
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       client_max_body_size 50M;
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       client_body_timeout 60;
       send_timeout 300;
       lingering_timeout 5;
       proxy_connect_timeout 90;
       proxy_send_timeout 300;
       proxy_read_timeout 90s;
       proxy_http_version 1.1;
       proxy_pass http://backend;
   }
3

@agnivade thank you for your reply.
I am using the nginx ingress controller inside of Kubernetes and have not made any chances to it or added any custom configuration.
I am therefore not able to edit the nginx config directly as you are describing.
On my experience this configuration should be covered by the helm chart itself.

4

Ah I see. I’ll ping some people in the github thread who would be able to help out.

5

I was able to resolve the issue with the help of some mattermost developers.
The main issue was the nginx ingress controller.

I think nginx.ingress.kubernetes.io annotation is for GitHub - kubernetes/ingress-nginx: Ingress-NGINX Controller for Kubernetes and the link you sent is GitHub - nginxinc/kubernetes-ingress: NGINX and NGINX Plus Ingress Controllers for Kubernetes which according to docs uses nginx.org annotation. I have no experience with the latter so it is hard for me to say if it is configured properly. (Comment on the github issue)

After installing the “standard” nginx controller (kubernetes/ingress-nginx), everything is working as expected.

6

Could you share your current configuration? Mine is kubernetes/ingress-nginx, but I am having severe websocket errors too that I can’t quite figure out. Of note, I’m using mattermost-operator. I had no real issues with the enterprise helm chart

7

Dear @Tarek ,
I am using the kubernetes/ingress-nginx with the standard helm installation and values.
You can follow the guide here and use the chart in the github repo. All configurations on my end are standard.

Have in mind that I am on a bare metal cluster with the metalLB load balancer. Your load balancer could also be a cause for your problems.