Infinite refresh loop from clients

Troubleshooting
1

Summary

A user who was previously logged in, and takes no action (does not log themselves out or anything like that), opens mattermost in a browser or desktop client and sees an infinite refresh loop. They may be able to access mattermost via a different client, but on the affected client they have no way to proceed.

Steps to reproduce

Observed over many mattermost versions; most recent occurrence on 4.3.2. Observed with Chrome and desktop clients on multiple OS. We delegate auth to gitlab. Other than that, no known way to reproduce.
I am guessing that it is triggered by the affected session reaching its timeout, so the session becomes logged out as expected. However I haven’t confirmed that.

Expected behavior

No infinite refresh loop :slight_smile:

Observed behavior

While stuck in the refresh loop, we can see the purple button for gitlab login appear on each render. Sometimes if the user clicks at the exact right moment they can reach the login screen and this resolves the issue.
We previously found that clearing all cache data in the browser/client also fixes it.
During the refresh loop, the log continuously fills with these messages:

[2017/11/16 11:43:44 AEDT] [EROR] /api/v4/users/me: code=401 rid=zoj1btgtspb7fgrjersbgbjwto uid= ip=10.70.34.172 Invalid or expired session, please login again. [details: UserReq
uired]
[2017/11/16 11:43:44 AEDT] [EROR] /api/v4/users/me/preferences: code=401 rid=xe1rx9gpotdp5qg3zm5xep54co uid= ip=10.70.34.172 Invalid or expired session, please login again. [deta
ils: UserRequired]
[2017/11/16 11:43:44 AEDT] [EROR] /api/v4/users/me/teams: code=401 rid=qor8umozofy45xzbnjym78z61c uid= ip=10.70.34.172 Invalid or expired session, please login again. [details: U
serRequired]
[2017/11/16 11:43:44 AEDT] [EROR] /api/v4/users/me/teams/members: code=401 rid=p8m4zwy71prypq4kceohxjtdsy uid= ip=10.70.34.172 Invalid or expired session, please login again. [de
tails: UserRequired]
[2017/11/16 11:43:44 AEDT] [EROR] /api/v4/users/me/teams/unread: code=401 rid=8i9ynzj4n7bc7nujrtic4kgqea uid= ip=10.70.34.172 Invalid or expired session, please login again. [det
ails: UserRequired]
[2017/11/16 11:43:44 AEDT] [EROR] /api/v4/emoji: code=401 rid=zhkph9gaufrbxpcnc8zs13dc9r uid= ip=10.70.34.172 Invalid or expired session, please login again. [details: UserRequir
ed]

Note: this issue was previously raised in [SOLVED] Infinite refresh loop in browser clients, with "Invalid or expired session" log error spam. I am raising a new topic because

  • the former one has been marked solved
  • I think the previous topic addressed some related but different behaviour for other users; I’d like to get back to the specific behaviour affecting my users
2

I’m seeing something similar here, Mattermost 4.4.1 loses session and users disappear but my clients don’t go into an infinite refresh. Seems like a bunch of bugs related to the session timing out.

3

Thanks @gubbins for the report.

  1. Can you let me know the values of the following settings in config.json, under ServiceSettings 
        "SessionLengthWebInDays"
        "SessionLengthMobileInDays"
        "SessionLengthSSOInDays"
        "SessionCacheInMinutes"
        "SessionIdleTimeoutInMinutes"
  1. Similarly, can you let me know the values for these config.json settings:
    "LocalizationSettings": {
        "DefaultServerLocale": "en",
        "DefaultClientLocale": "en",
        "AvailableLocales": ""
    },
  1. What languages do the users experiencing issues have set in Mattermost, do you know?

@congalong I also replied to you on the other thread :slight_smile:

4 
        "SessionLengthWebInDays": 90,
        "SessionLengthMobileInDays": 30,
        "SessionLengthSSOInDays": 90,
        "SessionCacheInMinutes": 10,

    "LocalizationSettings": {
        "DefaultServerLocale": "en",
        "DefaultClientLocale": "en",
        "AvailableLocales": ""
    },

All users are using English / en AFAIK.

5

Also, to touch on questions that came up before:

> select distinct Locale from Users;
+--------+
| Locale |
+--------+
| en     |
+--------+
1 row in set (0.01 sec)

We run mattermost behind nginx.

We delegate auth to GitLab Enterprise Edition 10.0.3-ee, set up as per the instructions. The only scope selected for the Application on the gitlab side is “api”.

6

Thanks @gubbins, very helpful. I’ll touch base with our engineers to get their thoughts on this.

7

Actually, @gubbins, if you can, please help upgrade to the latest Mattermost release, v4.4.1? https://about.mattermost.com/download/

I realize you’ve said this has reproduced over many Mattermost versions, but there were several bug fixes made on that release. I’ll still cross-post this to our engineers in the meantime.

8

OK done. It’s frustrating that I have no way to reproduce the issue (any suggestions?) but I will keep an eye on the logs.

9

Thanks @gubbins,

Let us know how it goes now that you’ve upgraded to v4.4.1

10

I’m getting the infinite loop. I got it after upgrading to 4.4.1. I seemed to happen after I logged off. It was happening in the web browsers, in the mac client, and the Windows client. The android app continued to work. Also, I was able to get the browsers to work after deleting all cookies. But I’m very frustrated with the Mac and PC client. Infinite loop won’t let me log on.

11

Thanks for your feedback @hpalmer,

Could you take a look at jasonblais questions above and provide the relevant responses for your set-up?

You might also find relevant information in this post

12

Lindy,

Attached is our details. Also, it seems to only be a problem with my account but it effects any workstation that I log into. Once I have logged on to a 3.7.1 client on a Mac or a PC and then log out the logon screen goes into an infinite loop.

I know the browser client uses a cookie. Does the full client use a cookie? I’d like to just delete my account and fix the workstations that are broken. We have tried to uninstall and reinstall the client but that does not fix the problem.

[Palmer, Harry70x93]

Harry J Palmer
Director of Information Systems
Charter Township of West Bloomfield
West Bloomfield, MI 48323
(Ph) 248.409.1585 (Cell) 248.390.6568

13

We used Revo and were able to completely remove Mattermost client. I looks like the files in …\appdata\roaming\ needed to be deleted to eliminate the infinite loop and the uninstaller didn’t do it.

We also delete the one problem user using the CLI command.

Everything is working fine for us now.

Harry Palmer

14

Hi @hpalmer,

Thanks for your feedback and confirming everything is working for you now! Great to hear :slight_smile:

Please feel free to let us know if you have further issues…

15

Hello
We are on 5.3.1 (We previoulsy were on 4.9, where the problem was also present. Updating didn’t change anything).
I have the same exact problem : People connected on day N will encounter an infinite loop page on day N+1.
The url in the adress bar of the browser shows a message about the user’s session (I will copy/paste it here when I encounter the problem again)

Trying to access the instance from another browser (or in private navigation) let you access the login page (but it seems you cannot actually log in, the process is stuck after you click the submit button)

The only fix I have for now is to restart the service via sudo systemctl restart mattermost

I made the updates mysfelf, and carefully read the release notes, but I can’t see that I missed anything. But maybe I did !

We have been running Mattermost for almost a year now, and we didn’t have any problem until last month (and it didn’t happened after any update)

Here is part of the log with the message We encountered an error finding the session:

{"level":"error","ts":1538722595.7574444,"caller":"jobs/jobs_watcher.go:70","msg":"Error occurred getting all pending statuses: SqlJobStore.GetAllByStatus: We couldn't get the jobs, Status=pending, context deadline exceeded"}
{"level":"info","ts":1538722604.8005764,"caller":"web/handlers.go:95","msg":"Invalid session","error":"SqlSessionStore.Get: We encountered an error finding the session, sessionIdOrToken=3g7ngmukx7dpmpgmx6sp1obj9c, context deadline exceeded"}
{"level":"error","ts":1538722604.8007221,"caller":"web/context.go:60","msg":"We encountered an error finding the session","path":"/error","request_id":"i6hfemjeob83jjyro3hj4eupzc","ip_addr":"81.250.181.127","user_id":"","method":"GET","err_where":"SqlSessionStore.Get","http_code":500,"err_details":"sessionIdOrToken=3g7ngmukx7dpmpgmx6sp1obj9c, context deadline exceeded"}
{"level":"info","ts":1538722605.6425412,"caller":"web/handlers.go:95","msg":"Invalid session","error":"SqlSessionStore.Get: We encountered an error finding the session, sessionIdOrToken=3g7ngmukx7dpmpgmx6sp1obj9c, context deadline exceeded"}
{"level":"error","ts":1538722605.6426735,"caller":"web/context.go:60","msg":"We encountered an error finding the session","path":"/","request_id":"6zc8qx8k4jbpzckz7t7ih8imer","ip_addr":"81.250.181.127","user_id":"","method":"GET","err_where":"SqlSessionStore.Get","http_code":500,"err_details":"sessionIdOrToken=3g7ngmukx7dpmpgmx6sp1obj9c, context deadline exceeded"}
{"level":"info","ts":1538722635.7982163,"caller":"web/handlers.go:95","msg":"Invalid session","error":"SqlSessionStore.Get: We encountered an error finding the session, sessionIdOrToken=3g7ngmukx7dpmpgmx6sp1obj9c, context deadline exceeded"}
{"level":"error","ts":1538722635.7983813,"caller":"web/context.go:60","msg":"We encountered an error finding the session","path":"/error","request_id":"p6ucajymejy4xxhfsp8c1u3ejr","ip_addr":"81.250.181.127","user_id":"","method":"GET","err_where":"SqlSessionStore.Get","http_code":500,"err_details":"sessionIdOrToken=3g7ngmukx7dpmpgmx6sp1obj9c, context deadline exceeded"}
{"level":"info","ts":1538722636.650756,"caller":"web/handlers.go:95","msg":"Invalid session","error":"SqlSessionStore.Get: We encountered an error finding the session, sessionIdOrToken=3g7ngmukx7dpmpgmx6sp1obj9c, context deadline exceeded"}
{"level":"error","ts":1538722636.6508753,"caller":"web/context.go:60","msg":"We encountered an error finding the session","path":"/","request_id":"cqyefz5713fb38cdsjpyzashyr","ip_addr":"81.250.181.127","user_id":"","method":"GET","err_where":"SqlSessionStore.Get","http_code":500,"err_details":"sessionIdOrToken=3g7ngmukx7dpmpgmx6sp1obj9c, context deadline exceeded"}

Can anybody help me find a way to fix this ?

Thanks !

16

Here is the url of the loopping page

https://talk.sopress.net/error?message=Nous+avons+rencontr%C3%A9+une+erreur+lors+de+la+recherche+de+la+session&s=MEUCIHxDa4wqkVg5V7uUMZRszUhAer_uxCI_kWFv841CmPT3AiEArtsC89-vT_cJF68SPkPt_L0W3GYkDs3rRymkwlP8_CA=

17

Hi @jilfransoi,

  1. Can you let me know the values of the following settings in config.json, under ServiceSettings 
        "SessionLengthWebInDays"
        "SessionLengthMobileInDays"
        "SessionLengthSSOInDays"
        "SessionCacheInMinutes"
        "SessionIdleTimeoutInMinutes"
  1. Similarly, can you let me know the values for these config.json settings:
    "LocalizationSettings": {
        "DefaultServerLocale": "en",
        "DefaultClientLocale": "en",
        "AvailableLocales": ""
    },
  1. Has this reproduced on the following:
    3a) Browsers (if so, which ones)?
    3b) Desktop app (if so, Mac, Windows and/or Linux)?
    3c) Mobile app (if so, iOS and/or Android)?
18

Hello, here are the settings :

    "SessionLengthWebInDays": 30,
    "SessionLengthMobileInDays": 30,
    "SessionLengthSSOInDays": 30,
    "SessionCacheInMinutes": 10,
    "SessionIdleTimeoutInMinutes": 365000,

I changed the SessionIdleTimeoutInMinutes value last week in an attempt to solve this problem. Before, it was set to 0 but the probleme happended all the same

"LocalizationSettings": {
    "DefaultServerLocale": "fr",
    "DefaultClientLocale": "fr",
    "AvailableLocales": "en,fr"
},

The problem occurs in the desktop app (Mac and Windows) and in browsers (tested only in chrome for now, but beginning to test it today in other browers)

It seems that the problem doesn’t occur in the mobile app (iOs & Android) but I am not 100% sure. Will try to have a definitive answer on that one.

Thanks you very much

19

@jilfransoi In the database, can you help check the Users table and see to it that all users’ Locale columns are not empty .

SELECT * FROM Users WHERE Locale="";

If there’s an empty Locale, then I suggests to set it to fr and observe if you still encounter the problem.

20

Hello and thank you for your help.

I launched your query but it returned no results. The behaviour of my instance has not changed for now.

I have settled for now to a daily restart of the service. I hope some day this bug will be resolved.