Summary
Certain users are marked as inactive every morning.
Steps to reproduce
-
Login through GitLab (CE 13.5.3)
-
Mattermost Version: 5.25.4
-
Database Schema Version: 5.25.0
-
Database: postgres
-
Session lengths are the default 30 days
Existing discussion in github: https://github.com/mattermost/mattermost-server/issues/16275
Expected behavior
Users should remain logged in as long as the session length is configured. Direct messages should not become archived just because the users has not logged-in recently.
Observed behavior
A small subset of users (~10) become marked as inactive every morning and all direct messages with them become archived. They can send messages but cannot receive any until they logout and login again. Once they re-login, their status changes to member in the system console, but their direct messages are archived until they send a message.
Note - the users do not have issues with their GitLab session length.
Note - this affects only people with names starting with A , B , C or D , so it looks like some user filtering going wrong.
Solutions Attempted:
- Revoke all sessions for the users and have them login fresh.
- I have provided some logs in the GitHub issue, and will gather other logs as needed.
Hello @itwasonlyabug! I read the GitHub issue, and I have to say, this is a very interesting situation that you are dealing with, I hope I’ll be able to contribute to helping this get resolved.
If I can ask, what are you using as your reverse proxy? (Ex. Apache, Nginx, etc)
In addition, can you confirm that you have reverse proxied the web socket as well? This is important, and something that I personally had issues with at first as well, because the web socket must have a separate proxy entry in the configuration files, otherwise it won’t be reverse proxied, because it is not a HTTP/HTTPS request.
Hi! We have NGINX, which is configured as per this page: https://docs.mattermost.com/install/install-ubuntu-1604.html#installing-nginx-server
We do not have customizations on top of those settings.
Alright, and are you using a secured HTTPS connection, and if so, what SSL protocol are you currently using?
Okay, so I just verified and we are not using HTTP/2 like shown in the docs, but the websocket and general location settings are the same.
For SSL we are using TLSv1.2.
After some testing, I think maybe all users with A and B letters are deactivated ( we don’t have that many) . I’ll verify this and reply.
1 Like
Alright, sounds good, I’ll be watching for the reply 
Confirmed, all users with names starting with A and B get deactivated each morning. Users with C & D names have stopped experiencing this issue.
1 Like
That is very intriguing to me, and certainly quite unique! @ahmaddanial I’m wondering, do you think this is more likely a server-side issue, or an in-transport/network issue? Just thinking of where would be best to start the debug and assessment process.
Hello, @itwasonlyabug
Users can get deactivated automatically based on the following circumstances in your case:
- Gitlab authentication is disabled from System Console.
- Through user or System Administrator interaction (for example, users are allowed to deactivate their account)
- Some sort of automation/provisioning from outside.
Since you were able to narrow down that the issue is now only happening for A&B users versus C&D, let us dive deeper to understand the differences between these two sets of user groups. Do you observe anything obvious between them?
Additionally, can the deactivated users still login to Gitlab based on the article I came across here?
Deactivating a user is functionally identical to blocking a user, with the following differences:
- It does not prohibit the user from logging back in via the UI.
- Once a deactivated user logs back into the GitLab UI, their account is set to active.
1 Like
Hi @ahmaddanial, here’s some information on the setup:
- The users are not deactivated/blocked in GitLab.
- The users can access GitLab just fine and their sessions are also preserved so they don’t have to login each morning.
- The issue appears only in Mattermost.
- The users also do not have this issue with pure AD authentication.
- We are only 2 users with System Admin permission in Mattermost and only I am actively administering it (I’ve checked :)) so nobody is deactivating the users manually.
- There is no custom automation added at any step between GitLab and Mattermost.
- We have started experiencing this issue in the past 6 months, there were no such problems before that. We haven’t really changed anything related to Mattermost in that time.
- I upgraded the Mattermost server 2 months ago from 5.5.0 to 5.25.4 (following the docs strictly) but I can verify that the issue started before that.
@XxLilBoPeepsxX For it to be happening only on the A-B users and not All users, I’m thinking something like session house-keeping or database-related. If it was transport / proxy I would expect it to be affecting more people with random names.
Alright, could you send your database schema of the Users table here, please?
It will look something like this, if you use the same commands that I did, which are also included below.
root@ubuntu:~# mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 10
Server version: 8.0.22-0ubuntu0.20.04.3 (Ubuntu)
Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> use mattermost
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show columns from Users;
+--------------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------------------+--------------+------+-----+---------+-------+
| Id | varchar(26) | NO | PRI | NULL | |
| CreateAt | bigint | YES | MUL | NULL | |
| UpdateAt | bigint | YES | MUL | NULL | |
| DeleteAt | bigint | YES | MUL | NULL | |
| Username | varchar(64) | YES | UNI | NULL | |
| Password | varchar(128) | YES | | NULL | |
| AuthData | varchar(128) | YES | UNI | NULL | |
| AuthService | varchar(32) | YES | | NULL | |
| Email | varchar(128) | YES | UNI | NULL | |
| EmailVerified | tinyint(1) | YES | | NULL | |
| Nickname | varchar(64) | YES | | NULL | |
| FirstName | varchar(64) | YES | | NULL | |
| LastName | varchar(64) | YES | | NULL | |
| Position | varchar(128) | YES | | NULL | |
| Roles | text | YES | | NULL | |
| AllowMarketing | tinyint(1) | YES | | NULL | |
| Props | text | YES | | NULL | |
| NotifyProps | text | YES | | NULL | |
| LastPasswordUpdate | bigint | YES | | NULL | |
| LastPictureUpdate | bigint | YES | | NULL | |
| FailedAttempts | int | YES | | NULL | |
| Locale | varchar(5) | YES | | NULL | |
| Timezone | text | YES | | NULL | |
| MfaActive | tinyint(1) | YES | | NULL | |
| MfaSecret | varchar(128) | YES | | NULL | |
+--------------------+--------------+------+-----+---------+-------+
25 rows in set (0.01 sec)
MySQL>
1 Like
@XxLilBoPeepsxX sorry for the late response, here it is (keep in mind that it’s from PSQL not MySQL):
Column | Type | Modifiers | Storage | Stats target | Description
--------------------------+-------------------------+---------------------------------------------------------------------------------------------------------+----------+--------------+-------------
id | character varying(26) | not null | extended | |
createat | bigint | | plain | |
updateat | bigint | | plain | |
deleteat | bigint | | plain | |
username | character varying(64) | | extended | |
password | character varying(128) | | extended | |
authdata | character varying(128) | | extended | |
authservice | character varying(32) | | extended | |
email | character varying(128) | | extended | |
emailverified | boolean | | plain | |
nickname | character varying(64) | | extended | |
firstname | character varying(64) | | extended | |
lastname | character varying(64) | | extended | |
position | character varying(128) | | extended | |
roles | character varying(256) | | extended | |
allowmarketing | boolean | | plain | |
props | character varying(4000) | | extended | |
notifyprops | character varying(2000) | | extended | |
lastpasswordupdate | bigint | | plain | |
lastpictureupdate | bigint | | plain | |
failedattempts | integer | | plain | |
locale | character varying(5) | | extended | |
mfaactive | boolean | | plain | |
mfasecret | character varying(128) | | extended | |
timezone | character varying(256) | default '{"automaticTimezone":"","manualTimezone":"","useAutomaticTimezone":"true"}'::character varying | extended | |
acceptedtermsofserviceid | character varying(64) | default ''::character varying | extended | |
Thanks for providing the schema, and letting me know it was PSQL, I took a look, it seems to all be correct, which leads me to narrow the issue down to either a network or API/WebSocket issue at this point. I’m going to look into my own installation to see if I can discover for myself the protocols and workflow of this function to see if I can tamper with it and figure a way to reproduce it out. Hopefully, it might work, I have no idea 
1 Like
Do you have an idea what kind / level of logging I can get that might shed some clarity? e.g. NGINX access logs, mattermost Trace logging?
I would suggest following the steps found here to capture logs, specifically the first step, which will write to the Nginx error log file, which will be easier/easiest to send here.
https://serverfault.com/a/435575
1 Like
@XxLilBoPeepsxX sorry for the delay, but this took some convincing of other people I got all logs that are not just 2xx codes, but it’s a big chunk, how should I best share them here? Or if you can give me something that I can filter for that would also narrow it down.
A trend I’m seeing in these logs is that it’s mostly Safari and AppleWebkit requests, and I know that some of the users are Mac users so in the meantime I’ll check if the others are as well.
1 Like
Your all good, I 100% understand the process that may have needed to take place to obtain the information. I would suggest either Pastebin, creating a .zip file with all the files in the archive, or something along those lines. Basically, we’re going to be looking for 500 errors, 30X’s and anything that looks out of the ordinary compared to the rest of the logs.
@XxLilBoPeepsxX thank you for your support on this issue! I got a rejection attaching the logs and I can’t do anything more to help debug this.
I’m closing the issue.
1 Like
Alright, I guess that’s it then! If you want to continue trying to get it figured out, or anything else comes up, feel free to send me a message, I’d be more than happy to work with you and/or your development & IT team to get whatever comes up resolved!
1 Like