Very slow /unread endpoint call when switching channels

Troubleshooting
1

Summary
Since updating to mattermost 6.0, whenever I switch channels in the app I have to wait from 3 to 15 seconds because of a slow /api/v4/users/<id>/channels/<id>/posts/unread?limit_after=30&limit_before=30&skipFetchThreads=false&collapsedThreads=false&collapsedThreadsExtended=false API call.

Steps to reproduce
On our installation, switching channels is sufficient to trigger this behavior.

Expected behavior
The channels should load faster.

Observed behavior
I noticed a weird thing, the response shows a bunch of messages that I have already read, as I don’t have any unread badges in my whole mattermost app.

Screenshot of chrome devtools:

image
image485Ă—525 50.3 KB

2

Opened a ticket for our team to take a look: [MM-39433] Performance issues reported on 6.0 - Mattermost

3

Hey @luca-nardelli - I am wondering if turning ServiceSettings.EnablePermalinkPreviews to false has any effect here?

We are internally looking into this. But I quickly wanted to rule out this factor.

4

Hey, sorry for the delay in getting back to you. It seems I cannot set this option from mmctl, is this intended?

Full command:

mmctl --local config set ServiceSettings.EnablePermalinkPreviews false
Error: selected path object is not valid

5

I am experiencing the same issue ever since upgrading to 6.0, set EnablePermalinkPreviews=false and rebooted. Problem remains. And like with @luca-nardelli it is the unread call I am waiting for:

image
image1151Ă—21 5.19 KB

@agnivade, what do I use with PermalinkPreviews=false? Should I set it back?

Currently on 6.0rc2 (rc3 after the settings change and now 6.0.2) running on Ubuntu 20/Docker

6

@luca-nardelli - Ensure that you are running the latest mmctl packaged with 6.0. If you are using that, then I’m not very sure why that’s happening. Are you able to set other config options?

@tomz - Thanks for the confirmation. You can set it back to true if you prefer. I was just trying to isolate if that feature was responsible in any way.

7

Ok, given that I don’t know what this flag does I better go back to the default=true in this case. :slight_smile:

8

Updated today to 6.1 and the issue is still present, it actually feels worse

image

This is when opening a private channel I have with myself (i.e. the scratch pad)

9

@luca-nardelli - Just to confirm here, you are seeing this with EnablePermalinkPreviews disabled?

10

Yeah, somehow the update to 6.1 “fixed” the mmctl issue I had previously reported.

I ran the following to apply the setting (without restarting mattermost)

mmctl --local config set ServiceSettings.EnablePermalinkPreviews false
mmctl --local config reload

By the way, reproducing the issue is not super easy, because it looks like the /unread call is cached for some time, and subsequent calls (i.e. accesses to the channel) by the same user return quickly, but I guess it expires after some hours.

I’ll leave the option disabled for some time to better see how it fares

11

@luca-nardelli / @tomz - I have another setting which I would like to run by you all.

Can you also check if ServiceSettings.EnableLinkPreviews is set to true and if so, set it to false and see if that helps?

EDIT: To clarify, set both ServiceSettings.EnableLinkPreviews and ServiceSettings.EnablePermalinkPreviews to false.

Thanks.

12 
~ $ mmctl --local config get ServiceSettings.EnableLinkPreviews
true
~ $ mmctl --local config set ServiceSettings.EnableLinkPreviews false
Value changed successfully
~ $ mmctl --local config get ServiceSettings.EnableLinkPreviews
false
~ $ mmctl --local config get ServiceSettings.EnablePermalinkPreviews
false
~ $

Tried setting that now, and it looks like that did the trick! Of course we don’t have link previews now, but the loading times seem much more consistent. In the screenshot below I was opening random channels and you you can probably see when I disabled that ^^. By the way, I don’t know if it helps, but the slowest channel of them all was my personal account channel/scratchpad, where I have a bunch of links I’m saving for later (which would make sense since the issue is linked to link previews).

image
image477Ă—719 97.7 KB

Thank you very much for the assistance

13

Thanks for the confirmation @luca-nardelli !

@tomz - please let us know if the above steps help. Thanks.

14

Just tried it on one of my instances and it does feel snappier.

1 Like
15

Great to hear that! We now have a fairly good grip on the issue. You can follow along the progress on the ticket here Very slow /unread endpoint call when switching channels - #2 by amy.blais.

16

When the endpoint is loading (up to 25 seconds) our Postgres logs are filled with the following statements:

2021-11-26 12:28:29.682 CET [32350] DETAIL:  key « (hash)=(2252281287) » already exists.                                                                                                                             
2021-11-26 12:28:29.682 CET [32350] INSTRUCTION :  insert into "linkmetadata" ("hash","url","timestamp","type","data") values ($1,$2,$3,$4,$5);                                                                      
2021-11-26 12:28:29.694 CET [31892] ERROR:  duplicate key for constraint « linkmetadata_pkey »

(sorry for the approximate translation)

The hashes value are related to links that are posted in the channel being loaded, so it’s most certainly related to the slow loading time. I turned off the link preview setting through the UI console, and we’ll see how it goes.