Mattermost self-hosted extremely slow, and No Server logs after 27 June

Summary
From System Concole>Server Logs, there is no logs after 27th June.

Steps to reproduce
Deleted Journalctl files, restarted postgres.
Go to System Console>Server Logs

Expected behavior
There should be server logs after that mentioned date as well, and Mattermost should not be working slow.

Observed behavior
mattermost.log ends at 27th June, where I had restarted the Mattermost service.

Token request failed error was reported on 27th June, due to which users were unable to sign in to Mattermost via Gitlab. I had thought this might be due to lack of space, which caused DB to go down. So I deleted files under /var/log/journal, and restarted Postgres". The token request was gone, but mattermost became slow.

Hi,

what version of mattermost are you using?
Can you show the “LogSettings” section of your config.json?
How do you start mattermost? Via systemd/systemctl? If so, can you see something in systemctl status mattermost?

Best,
Alex

My Mattermost installations seems to have slowed down a lot as well. Users are reporting that it will take about 20-30s for the login screen to load.

My logs work, but they don’t show any indication that something is wrong.

Same questions to you :slight_smile: What mattermost version are you using and what does your infrastructure look like? What database backend are you using and is there any software component in front of mattermost (nginx, apache, loadbalancer, etc.)?

I am currently using
Mattermost Version: 7.0.1
Database Schema Version: 85
Database: postgres
I recently upgraded to it after encountering the issues. The images wouldn’t load after that problem, so i took a backup and upgraded it. Is there by any chance the configs changed? I have setup Gitlab Auth, which responds Token error. {“timestamp”:“2022-07-15 16:23:08.588 +05:45”,“level”:“error”,“msg”:“Bad response from token request.”,“caller”:“web/context.go:105”,“path”:“/signup/gitlab/complete”,“request_id”:“w7dcxsb8pjyutnoddm9ee4ffrh”,“ip_addr”:“27.34.20.55”,“user_id”:“”,“method”:“GET”,“err_where”:“AuthorizeOAuthUser”,“http_code”:500,“err_details”:“response_body={"error":"invalid_grant","error_description":"The provided authorization grant is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client."}, status_code=400, error=”}
{“timestamp”:“2022-07-15 16:23:41.111 +05:45”,“level”:“error”,“msg”:“Bad response from token request.”,“caller”:“web/context.go:105”,“path”:“/signup/gitlab/complete”,“request_id”:“7hdaee38offzigks8h8yjjtsme”,“ip_addr”:“27.34.20.55”,“user_id”:“”,“method”:“GET”,“err_where”:“AuthorizeOAuthUser”,“http_code”:500,“err_details”:“response_body={"error":"invalid_grant","error_description":"The provided authorization grant is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client."}, status_code=400, error=”}

You would have to re-authenticate the authentication source (gitlab in this case) if you restored a backup.
You should disable Gitlab SSO, login with a local admin and resetup the authentication provider following the instructions here:

https://docs.mattermost.com/onboard/sso-gitlab.html

Mattermost Version: 7.0.1
Database: MariaDB
LoadBalancer: HAProxy w/ SPOE ModSec
HyperVisor: Proxmox

Mattermost: RHEL 8, 4 CPU, 8GB RAM, 200G 15K SAS
MariaDB: RHEL 8, 4 CPU, 8GB RAM, 250G SSD

Slow initial load began with version 7. Once the login screen or channel screen loads there is no performance issues.

1 Like

I think v7 introduced a few additional things that will be fetched upon the first login, I did also experience that on my mobile phone, but it’s far away from being around 20-30s so far, so maybe I do either have a lot less users/posts compared to your setup or it must be something else.

7.1.1 just got released today and it also contains a few performance improvements - might be worth a shot to see if it fixes this problem.

I will upgrade to the latest version and test. I do have a lot of posts due to alerts being pushed into mattermost.

Same here, but we’re using Data Retention Policies to get rid of older posts in such notification-only channels, because we found out that it’s not necessary for us to keep them longer than 4 weeks. Maybe this is also interesting for you (depending on your edition, I think this is an enterprise feature).

Yeah, as I am only using community edition. No dice on 7.1.1, that initial load still takes a few seconds locally. It’s mostly RU users that are seeing the 20-30s load times.

I’ve been doing some recon and noticed some 401 errors in the console:

The first four lines are just from the calls plugin - do you use Mattermost calls? You could try to disable it temporarily and check if this changes something with regards to the startup time.

The last line just says that a font is missing (this font is also missing on my setup, so it might not be one of the default options and maybe unrelated).

Unforunately still no go. It does perform pretty poorly:

I do see the same bad values here (even worse, since my server is further away from Vancouver).

Upon initial page load, all plugins are being loaded, even before the user can login - not sure if this is necessary, but the biggest Javascript consumes on my end here are the enabled plugins and I think you can get some speed if you disable them (if you do not need them):

This is something the developers would have to take a look at to see what can be done about the initial payload (gtmetrix has quite some suggestions), but I’m not sure if this is really an issue, because this test is synthetic and does not take caching into account. The first uncached load of such a big application will always take some time, but the second load is much faster because all the relevant scripts are already cached. On my system, the second load time is around 2s, which is OK in my opinion, but of course, it can always be faster :slight_smile:

Does the server itself behave slow when you work with it or is content loading after being logged in slow or laggy or are you just trying to reduce the page load for the initial load of the login screen?