Mattermost v5.22.0

Troubleshooting
1

Dear all,

I’ve upgraded Mattermost on-premise version from 5.21.0 to 5.22.0 and after upgrade, we are experiencing very slow responsiveness of Mattermost chat.

Also receiving lots of Error msgs:

{“level”:“error”,“ts”:1587177247.4630964,“caller”:“http/h2_bundle.go:4211”,“msg”:“http2: server connection error from :24958: connection error: PROTOCOL_ERROR”,“source”:“httpserver”}

anyone else also experiencing this?

Using: On-Premise Mattermost Enterprise E10 - v5.22.0

2

Are you using Advanced Permissions https://docs.mattermost.com/deployment/advanced-permissions.html?

3

Are you seeing very slow responsiveness when posting messages / changing channels / something else?

4

We are using advanced permissions with some minor modifications made on system scheme.

Changing channels at most. When channel is changed it takes 1-2 sec to show the chat and a few secs more to show pictures etc… This didn’t happend before on 5.21

Edit: Works better today, but very low user activity, will see tomorrow.

Latest Release: 5.21.0 - Investigating a potential permissions related issue in 5.22.0
Should we revert back to 5.21.0?

THank you!

5

We are investigating a potential issue related to permissions but there’s no evidence that your issue is related to that. I’ll ask our dev team to take a look at the log error you shared.

6

@dodolo, is there any more context to the error you listed above? In isolation, it’s going to be hard to interpret. If you can share more logs, even if privately via customer support, we may be able to help analyze the issue in more detail.

If you’re having performance issues, I’d encourage you to revert to v5.21.0 and determine if the problem goes away or something else is a factor.

7

Hi @dodolo, what database are you using? in my case I have serious performance issues when switching channels as well, most times it keeps on loading … forever and users need to switch channel again and go back to original to force loading messages, but in any case faster than 10 - 15 seconds. Note this happened for months so not something introduced in 5.22 in my case

In my case is some mysql queries apparently could be causing the problem as I found tons of slow queries reported (60s until mysql timeout them). You could check that.
Following tickets opened for that:
https://mattermost.atlassian.net/browse/MM-23369
https://mattermost.atlassian.net/browse/MM-24170
https://mattermost.atlassian.net/browse/MM-24171

8

Hi @dodolo

Is your Mattermost server behind a reverse proxy or load balancer? If so, which one?

The reason I ask is because this error is generated by the Golang HTTP server library, and according to the source code this is what protocol requires if the client can’t accept push promises. From line 4997 in http/h2_bundle.go:

// A client cannot push. Thus, servers MUST treat the receipt of a PUSH_PROMISE
// frame as a connection error (Section 5.4.1) of type PROTOCOL_ERROR

There are a few other places where the PROTOCOL_ERROR is raised without any additional information in that file - lines 5019, 5038, and 5070 - and they all seem related to the Mattermost server pushing information to the client.