Upgrade from 5.32.1 to 5.33.0 brings on strange behaviors in the environment, such as double posts, posts being attributed to the wrong person, and high resource utilization
Running v5.33.0 seems to have created issues. I reverted back to v5.32.1 and the environment has stabilized
Describe your issue in detail
There are 1000’s of these messages being logged per minute.
The only commit i can find in the mattermost-server Github that i think may be a factor is:
Same library, similar issue. How does this tie into Mattermost?
Would you be open to upgrading to v5.33.1 to see if it fixes the issue?
Sure thing. This must have JUST been released this morning. I did the upgrade at 5AM today, and 5.33.0 was the latest.
I had the same issue and someone popped 5.33.1 which fixed it.
Hey there @efreeman - apologies for the inconvenience. Yes 5.33.1 fixes the issue you are seeing.
I just deployed 5.33.1, and the bad file descriptor errors are gone. I am seeing a lot of these errors in the logs, but i don’t see anything out of ordinary from the point-of-view of the desktop application.
2021-03-23T06:58:10.352Z error http/h2_bundle.go:4215 http2: server connection error from 141.157.218.32:64168: connection error: PROTOCOL_ERROR {“source”: “httpserver”}
2021-03-23T06:58:10.352Z error http/h2_bundle.go:4215 http2: server connection error from 141.157.218.32:64168: connection error: PROTOCOL_ERROR {“source”: “httpserver”}
2021-03-23T06:58:16.144Z error http/h2_bundle.go:4215 http2: server connection error from 108.41.225.235:53722: connection error: PROTOCOL_ERROR {“source”: “httpserver”}
2021-03-23T06:58:47.667Z error http/h2_bundle.go:4215 http2: server connection error from 74.64.160.204:54363: connection error: PROTOCOL_ERROR {“source”: “httpserver”}
2021-03-23T06:58:48.195Z error http/h2_bundle.go:4215 http2: server connection error from 71.202.126.110:51378: connection error: PROTOCOL_ERROR {“source”: “httpserver”}
2021-03-23T06:58:53.220Z error http/h2_bundle.go:4215 http2: server connection error from 69.248.58.194:57399: connection error: PROTOCOL_ERROR {“source”: “httpserver”}
2021-03-23T06:58:53.220Z error http/h2_bundle.go:4215 http2: server connection error from 69.248.58.194:57399: connection error: PROTOCOL_ERROR {“source”: “httpserver”}
2021-03-23T06:59:06.760Z error http/h2_bundle.go:4215 http2: server connection error from 119.18.2.179:10739: connection error: PROTOCOL_ERROR {“source”: “httpserver”}
2021-03-23T06:59:11.394Z error http/h2_bundle.go:4215 http2: server connection error from 71.190.230.144:55136: connection error: PROTOCOL_ERROR {“source”: “httpserver”}
2021-03-23T06:59:18.434Z error http/h2_bundle.go:4215 http2: server connection error from 100.2.223.183:50846: connection error: PROTOCOL_ERROR {“source”: “httpserver”}
2021-03-23T06:59:25.251Z error http/h2_bundle.go:4215 http2: server connection error from 141.149.48.58:64020: connection error: PROTOCOL_ERROR {“source”: “httpserver”}
I see. Were you not getting these errors earlier? We have seen these errors in the past, and it is due to an issue with the Go standard library which was just recently fixed (i.e. not yet in a released Go version). But it shouldn’t have to do anything with websockets.
OK. Thank you all so much for the assistance! Our Mattermost server has stabilized now, and we’re all set on 5.33.1!
Hi.
We just upgraded from v5.31.3 to 5.33.3, and our log is full of these errors
error http/h2_bundle.go:4215 http2: server connection error from : connection error: PROTOCOL_ERROR {"source": "httpserver"}
I’d be very glad for some pointers on how to diagnose this issue.
As with the above post, we’re seeing no impact on the service (desktop client and web-UI both work fine), but the logs are being flooded.
Digging through the logs, turns out this issue has been going on since we upgraded to v5.24.3 in August last year…
Hi, everyone.
Sharing the post that relates to the source code from the Golang HTTP server lib that is related to the error that we see here - Mattermost v5.22.0 - #8 by paulrothrock
Might help to clarify the entries that we see in the mattermost.log here.
Hello @selsno - this is a minor bug in Go’s HTTP/2 implementation, which was fixed by one of our devs. It does not cause any functional defects, just an error log will be shown.
The fix will be included in Go 1.17 which is yet to be released. After which, we would need to include that in Mattermost.
Thanks @agnivade , we see this as well. I’ll keep an eye for Go 1.17 in the release notes, and keep updating as usual.
