# Mattermost unstable after server update 4.7.3 **URL:** \ **Category:** Troubleshooting\ **Created:** [March 15, 2018, 3:19am UTC](https://forum.mattermost.com/t/mattermost-unstable-after-server-update-4-7-3/4637 "2018-03-15T03:19:49Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Ju\_St](https://avatars.discourse-cdn.com/v4/letter/j/35a633/32.png) [@Ju\_St](https://forum.mattermost.com/u/Ju_St)\ **Post date:** [March 15, 2018, 3:19am UTC](https://forum.mattermost.com/t/mattermost-unstable-after-server-update-4-7-3/4637/1 "2018-03-15T03:19:49Z")
#### Summary Mattermost is unstable after update to 4.7.3 (smtp TLS connection isn’t working, but this might be a coincidence) - keeps on restarting every 15-30 minutes. #### Steps to reproduce Debian 9, Nginx, Mattermost Enterprise Version 4.7.2 (upgraded to 4.7.3) follow standard procedures to update as indicated in the guide #### Expected behavior Same steps worked flawless to upgrade to 4.7.2 #### Observed behavior systemctl status mattermost shows that the server is running, but is stuck on ‘pinging sql master database’ ( **edit** ). Nginx shows 502 Bad Gateway, then after aun unusally long time mattermost starts running, but keeps on shutting down 1-2 times an hour. ---
**Author:** ![paulrothrock](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.mattermost.com/paulrothrock/32/2967_2.png) [@paulrothrock](https://forum.mattermost.com/u/paulrothrock)\ **Post date:** [March 15, 2018, 7:59pm UTC](https://forum.mattermost.com/t/mattermost-unstable-after-server-update-4-7-3/4637/2 "2018-03-15T19:59:16Z")
Happy to help but first I have a couple questions: - What version of MySQL and Nginx are you using? - Is the MySQL server functioning normally? ---
**Author:** ![Ju\_St](https://avatars.discourse-cdn.com/v4/letter/j/35a633/32.png) [@Ju\_St](https://forum.mattermost.com/u/Ju_St)\ **Post date:** [March 16, 2018, 1:27am UTC](https://forum.mattermost.com/t/mattermost-unstable-after-server-update-4-7-3/4637/3 "2018-03-16T01:27:17Z")
Hello Paul, we’re using postgresql 9.6 and nginx 1.10.3 (both seem to work fine) my mistake it should say: pinging sql master database in the original post edit: we tried again this morning and the update worked now! The strange thing is that the whole server seems very unstable since we tried to update yesterday. The logs are full of this: [2018/03/16 08:47:41 +07] [INFO] Server is initializing… [2018/03/16 08:47:41 +07] [INFO] Pinging SQL master database [2018/03/16 08:47:41 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. [2018/03/16 08:48:00 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’ [2018/03/16 08:48:01 +07] [INFO] Server is initializing… [2018/03/16 08:48:01 +07] [INFO] Pinging SQL master database [2018/03/16 08:48:01 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. [2018/03/16 08:48:27 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’ [2018/03/16 08:48:27 +07] [INFO] Server is initializing… [2018/03/16 08:48:27 +07] [INFO] Pinging SQL master database [2018/03/16 08:48:27 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. [2018/03/16 08:48:55 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’ [2018/03/16 08:48:55 +07] [INFO] Server is initializing… [2018/03/16 08:48:55 +07] [INFO] Pinging SQL master database [2018/03/16 08:48:55 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. [2018/03/16 08:49:11 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’ [2018/03/16 08:49:11 +07] [INFO] Server is initializing… [2018/03/16 08:49:11 +07] [INFO] Pinging SQL master database [2018/03/16 09:37:47 +07] [INFO] Pinging SQL master database [2018/03/16 09:37:47 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. [2018/03/16 09:37:55 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’ [2018/03/16 09:37:55 +07] [INFO] Server is initializing… [2018/03/16 09:37:55 +07] [INFO] Pinging SQL master database [2018/03/16 09:37:55 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. [2018/03/16 09:38:06 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’ [2018/03/16 09:38:06 +07] [INFO] Server is initializing… [2018/03/16 09:38:06 +07] [INFO] Pinging SQL master database [2018/03/16 09:38:06 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. [2018/03/16 09:38:40 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’ [2018/03/16 09:38:40 +07] [INFO] Server is initializing… [2018/03/16 09:38:40 +07] [INFO] Pinging SQL master database [2018/03/16 09:38:40 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. [2018/03/16 09:38:45 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’ [2018/03/16 09:38:45 +07] [INFO] Server is initializing… [2018/03/16 09:38:45 +07] [INFO] Pinging SQL master database [2018/03/16 09:38:45 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. [2018/03/16 09:39:16 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’ [2018/03/16 09:39:16 +07] [INFO] Server is initializing… [2018/03/16 09:39:16 +07] [INFO] Pinging SQL master database [2018/03/16 09:39:16 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. [2018/03/16 09:39:20 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’ [2018/03/16 09:39:20 +07] [INFO] Server is initializing… [2018/03/16 09:39:20 +07] [INFO] Pinging SQL master database [2018/03/16 09:39:20 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. [2018/03/16 09:39:50 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’ [2018/03/16 09:39:50 +07] [INFO] Server is initializing… [2018/03/16 09:39:50 +07] [INFO] Pinging SQL master database [2018/03/16 09:39:50 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. [2018/03/16 09:39:55 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’ [2018/03/16 09:39:55 +07] [INFO] Server is initializing… [2018/03/16 09:39:55 +07] [INFO] Pinging SQL master database [2018/03/16 09:39:55 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. [2018/03/16 09:40:32 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’ [2018/03/16 09:40:32 +07] [INFO] Server is initializing… [2018/03/16 09:40:32 +07] [INFO] Pinging SQL master database [2018/03/16 09:40:32 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. [2018/03/16 09:40:36 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’ [2018/03/16 09:40:36 +07] [INFO] Server is initializing… [2018/03/16 09:40:36 +07] [INFO] Pinging SQL master database [2018/03/16 09:40:36 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. [2018/03/16 09:41:15 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’ [2018/03/16 09:41:15 +07] [INFO] Server is initializing… [2018/03/16 09:41:15 +07] [INFO] Pinging SQL master database [2018/03/16 09:41:16 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. [2018/03/16 09:41:20 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’ [2018/03/16 09:41:20 +07] [INFO] Server is initializing… [2018/03/16 09:41:20 +07] [INFO] Pinging SQL master database [2018/03/16 09:41:20 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. [2018/03/16 09:41:50 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’ [2018/03/16 09:41:50 +07] [INFO] Server is initializing… [2018/03/16 09:41:50 +07] [INFO] Pinging SQL master database [2018/03/16 09:41:50 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. [2018/03/16 09:41:54 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’ [2018/03/16 09:41:54 +07] [INFO] Server is initializing… [2018/03/16 09:41:54 +07] [INFO] Pinging SQL master database [2018/03/16 09:41:54 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. [2018/03/16 09:42:27 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’ [2018/03/16 09:42:27 +07] [INFO] Server is initializing… [2018/03/16 09:42:27 +07] [INFO] Pinging SQL master database [2018/03/16 09:42:27 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. [2018/03/16 09:42:32 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’ [2018/03/16 09:42:32 +07] [INFO] Server is initializing… [2018/03/16 09:42:32 +07] [INFO] Pinging SQL master database [2018/03/16 09:42:32 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. [2018/03/16 09:42:54 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’ [2018/03/16 09:42:54 +07] [INFO] Server is initializing… [2018/03/16 09:42:54 +07] [INFO] Pinging SQL master database ---
**Author:** ![Ju\_St](https://avatars.discourse-cdn.com/v4/letter/j/35a633/32.png) [@Ju\_St](https://forum.mattermost.com/u/Ju_St)\ **Post date:** [March 19, 2018, 4:17am UTC](https://forum.mattermost.com/t/mattermost-unstable-after-server-update-4-7-3/4637/4 "2018-03-19T04:17:28Z")
We’ve upgraded to 4.8.0 today, which also attempted (and succeeded) in upgrading the database server. Seems to be stable again now. Will report back if problems persist. ---
**Author:** ![lindy65](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.mattermost.com/lindy65/32/1811_2.png) [@lindy65](https://forum.mattermost.com/u/lindy65)\ **Post date:** [March 22, 2018, 8:38am UTC](https://forum.mattermost.com/t/mattermost-unstable-after-server-update-4-7-3/4637/5 "2018-03-22T08:38:07Z")
Thanks @Ju_St, Let us know how it goes and whether you need any help troubleshooting further… ---
**Author:** ![Ju\_St](https://avatars.discourse-cdn.com/v4/letter/j/35a633/32.png) [@Ju\_St](https://forum.mattermost.com/u/Ju_St)\ **Post date:** [March 28, 2018, 3:20am UTC](https://forum.mattermost.com/t/mattermost-unstable-after-server-update-4-7-3/4637/6 "2018-03-28T03:20:29Z")
Hello lindy, the server runs stable most of the time, but weirdly some random actions cause a restart. Most recent example is uploading a certain image (8.4 MB), which causes a restart with the logs not showing any details on why it restarted. It just shows this: [2018/03/28 10:15:39 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’ [2018/03/28 10:15:39 +07] [INFO] Server is initializing… [2018/03/28 10:15:39 +07] [INFO] Pinging SQL master database [2018/03/28 10:15:40 +07] [INFO] Current version is 4.8.0 (4.8.0/Thu Mar 15 23:11:12 UTC 2018/d1ed79a66c7930020b7fda046a11050f5890c84e/6f563c1fa497d123134fd499eb37674eb2511325) [2018/03/28 10:15:40 +07] [INFO] Enterprise Enabled: true [2018/03/28 10:15:40 +07] [INFO] Current working directory is /opt/mattermost [2018/03/28 10:15:40 +07] [INFO] Loaded config file from /opt/mattermost/config/config.json [2018/03/28 10:15:40 +07] [INFO] Connection to S3 or minio is good. Bucket exists. [2018/03/28 10:15:40 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. [2018/03/28 10:15:40 +07] [INFO] Starting up plugins [2018/03/28 10:15:40 +07] [WARN] unable to create user namespace: fork/exec /proc/self/exe: permission denied [2018/03/28 10:15:40 +07] [WARN] plugin sandboxing is not supported. plugins will run with the same access level as the server [2018/03/28 10:15:41 +07] [INFO] Starting Server… [2018/03/28 10:15:41 +07] [INFO] Server is listening on XXX.XXX.XXX.XXX:8065 [2018/03/28 10:15:41 +07] [INFO] API version 3 is scheduled for deprecation. Please see [https://api.mattermost.com](https://api.mattermost.com) for details. [2018/03/28 10:15:41 +07] [INFO] Starting 2 websocket hubs [2018/03/28 10:15:41 +07] [INFO] Starting workers [2018/03/28 10:15:41 +07] [INFO] Starting schedulers. Adding new users with the platform command also seems to cause a restart (although the users are successfully created, while the file upload is stuck indefinitely) ---
**Author:** ![lindy65](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.mattermost.com/lindy65/32/1811_2.png) [@lindy65](https://forum.mattermost.com/u/lindy65)\ **Post date:** [March 29, 2018, 7:45am UTC](https://forum.mattermost.com/t/mattermost-unstable-after-server-update-4-7-3/4637/7 "2018-03-29T07:45:29Z")
Thanks for the feedback @Ju_St, I’ve asked our engineers to help troubleshoot this issue… ---
**Author:** ![jwilander](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.mattermost.com/jwilander/32/8440_2.png) [@jwilander](https://forum.mattermost.com/u/jwilander)\ **Post date:** [March 29, 2018, 3:15pm UTC](https://forum.mattermost.com/t/mattermost-unstable-after-server-update-4-7-3/4637/8 "2018-03-29T15:15:45Z")
@Ju_St couple questions to help troubleshoot: 1. How large of a deployment do you have? Some good numbers would be number of user and channel rows in your database 2. Are you able to capture the stderr output of the Mattermost process? Sometimes that has more information about the issue ---
**Author:** ![Ju\_St](https://avatars.discourse-cdn.com/v4/letter/j/35a633/32.png) [@Ju\_St](https://forum.mattermost.com/u/Ju_St)\ **Post date:** [March 30, 2018, 2:48am UTC](https://forum.mattermost.com/t/mattermost-unstable-after-server-update-4-7-3/4637/9 "2018-03-30T02:48:27Z")
Hello jwilander, we currently have 192 channel rows and 113 user rows. Let me check to see if I can get the stderr output. ---
**Author:** ![jwilander](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.mattermost.com/jwilander/32/8440_2.png) [@jwilander](https://forum.mattermost.com/u/jwilander)\ **Post date:** [April 2, 2018, 12:10pm UTC](https://forum.mattermost.com/t/mattermost-unstable-after-server-update-4-7-3/4637/10 "2018-04-02T12:10:11Z")
Ok with that low of row counts that rules out it being related to a performance issue we were having. Let me know if you’re able to grab the stderr ---
**Author:** ![Ju\_St](https://avatars.discourse-cdn.com/v4/letter/j/35a633/32.png) [@Ju\_St](https://forum.mattermost.com/u/Ju_St)\ **Post date:** [April 5, 2018, 2:09am UTC](https://forum.mattermost.com/t/mattermost-unstable-after-server-update-4-7-3/4637/11 "2018-04-05T02:09:34Z")
Sorry, no luck so far with the stderr output. Any other things I could try to get more information about the issue? ---
**Author:** ![Ju\_St](https://avatars.discourse-cdn.com/v4/letter/j/35a633/32.png) [@Ju\_St](https://forum.mattermost.com/u/Ju_St)\ **Post date:** [April 9, 2018, 2:08am UTC](https://forum.mattermost.com/t/mattermost-unstable-after-server-update-4-7-3/4637/12 "2018-04-09T02:08:28Z")
Another thing I noticed, not sure if this is supposed to happen: Whenever I create user accounts through the platform commands it re-initiates the server **For example these two commands:** sudo ./platform user create --email [xxxxx@xxxx.com](mailto:xxxxx@xxxx.com) --username xxxxx --password xxxx sudo ./platform user verify [xxxxx@xxxx.com](mailto:xxxxx@xxxx.com) **Show this in the logs for each user created this way:** [2018/04/09 09:04:00 +07] [INFO] Server is initializing… [2018/04/09 09:04:00 +07] [INFO] Pinging SQL master database [2018/04/09 09:04:00 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. **And then a bit later:** [2018/04/09 09:09:07 +07] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’ [2018/04/09 09:09:07 +07] [INFO] Server is initializing… [2018/04/09 09:09:07 +07] [INFO] Pinging SQL master database [2018/04/09 09:09:09 +07] [INFO] Current version is 4.8.0 (4.8.0/Thu Mar 15 23:11:12 UTC 2018/d1ed79a66c7930020b7fda046a11050f5890c84e/6f563c1fa497d123134fd499eb37674eb2511325) [2018/04/09 09:09:09 +07] [INFO] Enterprise Enabled: true [2018/04/09 09:09:09 +07] [INFO] Current working directory is /opt/mattermost [2018/04/09 09:09:09 +07] [INFO] Loaded config file from /opt/mattermost/config/config.json [2018/04/09 09:09:09 +07] [INFO] Connection to S3 or minio is good. Bucket exists. [2018/04/09 09:09:09 +07] [INFO] License key from [https://mattermost.com](https://mattermost.com) required to unlock enterprise features. [2018/04/09 09:09:09 +07] [INFO] Starting up plugins [2018/04/09 09:09:09 +07] [WARN] unable to create user namespace: fork/exec /proc/self/exe: permission denied [2018/04/09 09:09:09 +07] [WARN] plugin sandboxing is not supported. plugins will run with the same access level as the server [2018/04/09 09:09:10 +07] [INFO] Starting Server… [2018/04/09 09:09:10 +07] [INFO] Server is listening on XXX.XXX.XXX.XXX:8065 [2018/04/09 09:09:10 +07] [INFO] API version 3 is scheduled for deprecation. Please see [https://api.mattermost.com](https://api.mattermost.com) for details. [2018/04/09 09:09:10 +07] [INFO] Starting 2 websocket hubs [2018/04/09 09:09:10 +07] [INFO] Starting workers [2018/04/09 09:09:10 +07] [INFO] Starting schedulers. The users are created and work, but should the server initialize again each time? ---
**Author:** ![jwilander](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.mattermost.com/jwilander/32/8440_2.png) [@jwilander](https://forum.mattermost.com/u/jwilander)\ **Post date:** [April 13, 2018, 12:20pm UTC](https://forum.mattermost.com/t/mattermost-unstable-after-server-update-4-7-3/4637/13 "2018-04-13T12:20:41Z")
Seeing: ```auto [2018/04/09 09:04:00 +07] [INFO] Server is initializing… [2018/04/09 09:04:00 +07] [INFO] Pinging SQL master database [2018/04/09 09:04:00 +07] [INFO] License key from https://mattermost.com required to unlock enterprise features. ``` after running a platform command is normal but it shouldn’t cause an entire server restart a few minutes later. One thing you might be able to try is manually running the server attached at the command line with `./bin/platform`. If you can catch a server restart that way it should also print out the stderr if it’s being caused a by a crash ---
**Author:** ![Ju\_St](https://avatars.discourse-cdn.com/v4/letter/j/35a633/32.png) [@Ju\_St](https://forum.mattermost.com/u/Ju_St)\ **Post date:** [April 20, 2018, 8:51am UTC](https://forum.mattermost.com/t/mattermost-unstable-after-server-update-4-7-3/4637/14 "2018-04-20T08:51:09Z")
It seems like updating to 4.9 has resolved the issue. I just tried to upload files when starting the server from the platform command and from the normal systemctl start command. In both cases uploading the files worked without a server crash/restart. One thing I saw when starting the server from the platform command was this (over and over again) though: > [2018/04/20 15:41:09 +07] [EROR] /api/v4/plugins/webapp:GetActivePluginManifests code=501 rid=m5i7o7nhbfguzenwg1bcrnuyyo uid=8oyg7kc89tfjbrbpmb7hos173y ip=removed.ip.address Plugins have been disabled. Please check your logs for details. [details:] > [2018/04/20 15:41:47 +07] [EROR] /api/v4/plugins/webapp:GetActivePluginManifests code=501 rid=n1ezh7bciirztpxuhntp5eit6w uid=ej6n4ptsc7no5jkthdhaxqeufr ip=removed.ip.address Plugins have been disabled. Please check your logs for details. [details:] > [2018/04/20 15:41:47 +07] [EROR] /api/v4/plugins/webapp:GetActivePluginManifests code=501 rid=zk55okuehtrydcdqtex3oe9fdr uid=8ugq1t7363fhfq5hkijumfrp3a ip=removed.ip.address Plugins have been disabled. Please check your logs for details. [details:] > [2018/04/20 15:42:42 +07] [EROR] /api/v4/plugins/webapp:GetActivePluginManifests code=501 rid=ra3p1o4ccjyf8nygd3d4hiueze uid=4xa5eyw3pt83zch8hw6ba5dhzo ip=removed.ip.address Plugins have been disabled. Please check your logs for details. [details:] > [2018/04/20 15:42:55 +07] [EROR] /api/v4/plugins/webapp:GetActivePluginManifests code=501 rid=8ww3e1anijdo9dw77qdequtj1w uid=zywk4d55wtgzmkmjrm4ny1mcao ip=removed.ip.address Plugins have been disabled. Please check your logs for details. [details:] > [2018/04/20 15:43:02 +07] [EROR] /api/v4/plugins/webapp:GetActivePluginManifests code=501 rid=bdcgehanb7rs3jefnrhxthjauh uid=uq1dehr39tbkig51dd38jzqe8r ip=removed.ip.address Plugins have been disabled. Please check your logs for details. [details:]