Since MM 5.33 the http version needs to be v1.1.
We have set proxy_http_version 1.1; in our NGINX proxy in the server{} block.
However we still see the following warning in the logs:
The HTTP version field was detected as 1.0 during WebSocket handshake. This is most probably due to an incorrect proxy configuration. Please upgrade your proxy config to set the header version to a minimum of 1.1."
Is the configuration issue on our side or is this a false-positive in mattermost?
We are working on a dot release to fix it Mattermost Changelog — Mattermost 5.33 documentation. Installing a reverse proxy (Nginx) may help in the meantime
@amy.blais Thanks.
What do you mean by “install a reverse prox may help?”
We already have a NGINX running in front of MM with the respective setting set (as mentioned in my OP).
Has everything been configured as per the docs? Depending on which guide you’ve followed, e.g.
Hello! Sorry to hear you’re having issues, could you perhaps send us your Nginx reverse proxy configuration file here, so I can take a look at it? I’m fairly certain that this requires just a few small changes to the Nginx configuration file and the enabling of a module.
Also, FYI v5.33.1 has now been released.
Thanks @amy.blais and @XxLilBoPeepsxX.
I’ve updated to 5.33.1 and double-checked the NGINX config as stated in your docs.
We still see the warning above in the logs.
Here’s our config (with 172.17.0.1 being the docker0 gateway)
# This block is useful for debugging TLS v1.3. Please feel free to remove this
# and use the `$ssl_early_data` variable exposed by nginx directly should you
# wish to do so.
map $ssl_early_data $tls1_3_early_data {
"~." $ssl_early_data;
default "";
}
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;
server {
listen 443 ssl http2;
[...] # other settings
proxy_http_version 1.1;
location ~ /mattermost/api/v[0-9]+/(users/)?websocket$ {
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
client_max_body_size 50M;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_buffers 256 16k;
proxy_buffer_size 16k;
client_body_timeout 60;
send_timeout 300;
lingering_timeout 5;
proxy_http_version 1.1;
proxy_connect_timeout 90;
proxy_send_timeout 300;
proxy_read_timeout 90s;
proxy_pass http://172.17.0.1:8065;
}
location /mattermost {
client_max_body_size 50M;
proxy_set_header Connection "";
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_buffers 256 16k;
proxy_buffer_size 16k;
proxy_read_timeout 600s;
proxy_cache mattermost_cache;
proxy_cache_revalidate on;
proxy_cache_min_uses 2;
proxy_cache_use_stale timeout;
proxy_cache_lock on;
proxy_http_version 1.1;
proxy_pass http://172.17.0.1:8065/mattermost;
}
}
That is indeed strange @pat-s. Would you be able to share with us a tcpdump of the incoming traffic to the Mattermost instance? Please feel free to DM me directly at @agniva.de.sarker on community.mattermost.com. I want to make sure there is nothing else that is changing the header here.
The Mattermost code just checks the HTTP header version, and if it detects it to be 1.0, it logs this message. Therefore, it seems a bit weird to have this log even after correcting the config.
I believe I have found at least one potential issue!
You have, from what it appears, set your server to be listening on port 443 with the HTTP2 protocol, and then directly underneath that configuration line, it appears that you tell the reverse proxy to use HTTP Version 1.1. I’m going to have to do a bit of research to verify the incompatibility, but I am under the impression that you’re generally supposed to pick one or the other, and that using both on the same host/virtualhost doesn’t work.
Thanks!
Should proxy_http_version 1.1; only be set in the “location” block then (we’ve had that before, I added proxy_http_version 1.1 to the 443 server block after the I saw the log entries)? Or only in server{} which listens on port 80?
I left out the port 80 part. We are using the common 80 → 443 setup.
I think I would suggest trying changing your HTTP2 to the HTTP 1.1 protocol, based of the research I did, and this Apache information page, which provides some useful insight. (Yes, I’m aware that you are using NGINX, however these information bits are universal regardless of where they are obtained from:) )
Hi !
I have also the same problem, but it is for plugins (grumble plugin)
Here is what it said : "The HTTP version field was detected as 1.0 during WebSocket handshake. This is most probably due to an incorrect proxy configuration. Please upgrade your proxy config to set the header version to a minimum of 1.1."}
That is the same thing, but I’m running Mattermost on Ubuntu 20.04 LTS, so I don’t have the same config for nginx.
Here is my config :
server {
listen 443 ssl http2;
server_name mattermost.domain.com ;
ssl on;
ssl_certificate /etc/letsencrypt/live/mattermost.domain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/mattermost.domain.com/privkey.pem;
ssl_session_timeout 1d;
ssl_protocols TLSv1.2;
ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA->
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:50m;
# HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
add_header Strict-Transport-Security max-age=15768000;
# OCSP Stapling ---
# fetch OCSP records from URL in ssl_certificate and cache them
ssl_stapling on;
ssl_stapling_verify on;
location / {
client_max_body_size 5000M;
proxy_set_header Connection "";
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_buffers 256 16k;
proxy_buffer_size 16k;
proxy_read_timeout 600s;
proxy_cache mattermost_cache;
proxy_cache_revalidate on;
proxy_cache_min_uses 2;
proxy_cache_use_stale timeout;
proxy_cache_lock on;
proxy_http_version 1.1;
proxy_pass http://localhost:8065;
}
location ~ /api/v[0-9]+/(users/)?websocket$ {
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
client_max_body_size 500M;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_buffers 256 16k;
proxy_buffer_size 16k;
client_body_timeout 60;
send_timeout 300;
lingering_timeout 5;
proxy_connect_timeout 90;
proxy_send_timeout 300;
proxy_read_timeout 90s;
proxy_pass http://localhost:8065;
}
Can anyone tell what’s wrong ?
I think I would suggest trying changing your HTTP2 to the HTTP 1.1 protocol, based of the research I did, and this Apache information page, which provides some useful insight. (Yes, I’m aware that you are using NGINX, however these information bits are universal regardless of where they are obtained from:) )
I am not sure if using HTTP 1.1 is better than HTTP2 - given that this would apply to all services in our NGINX then. Isn’t HTTP2 supposed to be safer and faster?
I am not an expert in this area though.
In addition, with MM v5.33.2 the HTTP_VERSION change was reverted and my logs are now happy again, so I guess I’ll keep it as is.
The one thing I notice about this is that there is not a specification in this particular block (which contains the websocket proxy information) of the HTTP version with a line such as proxy_http_version 1.1;, I’m not 100% sure if that would solve the issue as I’m an expert with Apache, and while I am able to use Nginx extremely well, it’s not second nature to me as Apache is, but regardless, it can’t hurt to try, right?
Your correct, HTTP2 is, in theory, safer, faster, and more secure than the plain text HTTP1 alternative, when applied correctly, but since the release of the HTTP2 standard, the depreciation of HTTP1 has still not been made across the board, so both are still in use in some applications/websites, and likewise.
Regardless, I’m glad to hear everything seems to be working!
Hi !
Thank you for your answer !
Non hopefully, it didn’t worked…
still the same {"level":"warn","ts":1617034091.3143902,"caller":"api4/websocket.go:38","msg":"The HTTP version field was detected as 1.0 during WebSocket handshake. This is most probably due to an incorrect proxy configuration. Please upgrade your proxy config to set the header version to a minimum of 1.1."}
I took the time to research the Official Nginx Proxy Configuration Documentation and compared the contents of the sample file that they provided on that webpage to the configuration that you provided (you can view this comparison here) and it seems like there is a lot that is missing, I’m wondering if this could be contributing to the issues your having.
@XxLilBoPeepsxX Thanks!
I left most configs from the server block out but the location blocks should be pretty much equal.
Thanks for doing the diff but I think given that I posted a stripped version, it it not so easy to read.
Our instance was (and again is with 5.33.2) running fine without any issues or log entries.
So other than the warning in the logs, is everything still functioning properly and as expected?