Mattermost default plugins not visible after upgrade

Troubleshooting
1

Hi,
I tried to update our Mattermost production istance on Centos 7.6 (in VMWare environment) passing from 4.5 version to 5.16.3 following ufficial upgrade guide:

https://docs.mattermost.com/administration/upgrade.html

The upgrade itself works, but prepackaged plugins are not visible from system console, neither enabling or disabling plugins selecting right flag and saving…
I tried to googling for a solution and do the upgrade progress twice on a test environment clone of the production, but the result is always the same.
Could you please help me find a solution?

Looking at “server logs” into system console I found the message “Plugins have been disabled”:

{“level”:“info”,“ts”:1574068513.593584,“caller”:“mlog/log.go:166”,“msg”:“Invalid session”,“error”:“GetSession: Invalid session token=ozt49hkikbyn7gypdah19rkp1w, err=, “}
{“level”:“info”,“ts”:1574068513.5997508,“caller”:“mlog/log.go:166”,“msg”:“Invalid session”,“error”:“GetSession: Invalid session token=ozt49hkikbyn7gypdah19rkp1w, err=, “}
{“level”:“info”,“ts”:1574068513.8368235,“caller”:“mlog/log.go:166”,“msg”:“Invalid session”,“error”:“GetSession: Invalid session token=ozt49hkikbyn7gypdah19rkp1w, err=, “}
{“level”:“info”,“ts”:1574068513.838884,“caller”:“mlog/log.go:166”,“msg”:“Invalid session”,“error”:“GetSession: Invalid session token=ozt49hkikbyn7gypdah19rkp1w, err=, “}
{“level”:“info”,“ts”:1574068513.8433416,“caller”:“mlog/log.go:166”,“msg”:“Invalid session”,“error”:“GetSession: Invalid session token=ozt49hkikbyn7gypdah19rkp1w, err=, “}
{“level”:“info”,“ts”:1574068513.8435428,“caller”:“mlog/log.go:166”,“msg”:“Invalid session”,“error”:“GetSession: Invalid session token=ozt49hkikbyn7gypdah19rkp1w, err=, “}
{“level”:“info”,“ts”:1574068513.844091,“caller”:“mlog/log.go:166”,“msg”:“Invalid session”,“error”:“GetSession: Invalid session token=ozt49hkikbyn7gypdah19rkp1w, err=, “}
{“level”:“error”,“ts”:1574068514.0615156,“caller”:“mlog/log.go:174”,“msg”:“Plugins have been disabled. Please check your logs for details.”,“path”:”/api/v4/plugins/webapp”,“request_id”:“7jegaxs7r7n8fkwp81fmps695e”,“ip_addr”:“172.16.20.99”,“user_id”:””,“method”:“GET”,“err_where”:“GetActivePluginManifests”,“http_code”:501,“err_details”:””}
{“level”:“error”,“ts”:1574068528.7409122,“caller”:“mlog/log.go:174”,“msg”:“Plugins have been disabled. Please check your logs for details.”,“path”:”/api/v4/plugins”,“request_id”:“nr6fmg8e5fr79e7oiscnkyg35r”,“ip_addr”:“172.16.20.99”,“user_id”:“1c49g3xy9fr7zx6kjhdd41exmc”,“method”:“GET”,“err_where”:“GetPlugins”,“http_code”:501,“err_details”:””}
{“level”:“error”,“ts”:1574068538.5922909,“caller”:“mlog/log.go:174”,“msg”:“Plugins have been disabled. Please check your logs for details.”,“path”:”/api/v4/plugins/statuses”,“request_id”:“87cw6mu5cp8dtfiw79u86ahxea”,“ip_addr”:“172.16.20.99”,“user_id”:“1c49g3xy9fr7zx6kjhdd41exmc”,“method”:“GET”,“err_where”:“GetPluginStatuses”,“http_code”:501,“err_details”:””}
{“level”:“error”,“ts”:1574069253.54626,“caller”:“mlog/log.go:174”,“msg”:“Plugins have been disabled. Please check your logs for details.”,“path”:"/api/v4/plugins/statuses",“request_id”:“6a44f8zcgjfspdm45174knpe4y”,“ip_addr”:“172.16.20.99”,“user_id”:“1c49g3xy9fr7zx6kjhdd41exmc”,“method”:“GET”,“err_where”:“GetPluginStatuses”,“http_code”:501,“err_details”:""}
{“level”:“error”,“ts”:1574069256.8254158,“caller”:“mlog/log.go:174”,“msg”:“An error occurred saving the configuration”,“path”:"/api/v4/config",“request_id”:“ny14r75mhiydtdzmsjeui4iuco”,“ip_addr”:“172.16.20.99”,“user_id”:“1c49g3xy9fr7zx6kjhdd41exmc”,“method”:“PUT”,“err_where”:“saveConfig”,“http_code”:500,“err_details”:“failed to persist: failed to write file: open /data/mattermost/versions/mattermost-4.5.0/config/config.json: permission denied”}
{“level”:“error”,“ts”:1574069260.660455,“caller”:“mlog/log.go:174”,“msg”:“An error occurred saving the configuration”,“path”:"/api/v4/config",“request_id”:“jt3foiw9n7d1d8neixy5cwjpxe”,“ip_addr”:“172.16.20.99”,“user_id”:“1c49g3xy9fr7zx6kjhdd41exmc”,“method”:“PUT”,“err_where”:“saveConfig”,“http_code”:500,“err_details”:“failed to persist: failed to write file: open /data/mattermost/versions/mattermost-4.5.0/config/config.json: permission denied”}
{“level”:“error”,“ts”:1574069274.0038836,“caller”:“mlog/log.go:174”,“msg”:“Plugins have been disabled. Please check your logs for details.”,“path”:"/api/v4/plugins/statuses",“request_id”:“qt7iiqtuktgtjb4jeqgfs533sa”,“ip_addr”:“172.16.20.99”,“user_id”:“1c49g3xy9fr7zx6kjhdd41exmc”,“method”:“GET”,“err_where”:“GetPluginStatuses”,“http_code”:501,“err_details”:""}

Thanks in advance
Marco

2

@mbarbaro Can you help with your settings from here https://docs.mattermost.com/administration/config-settings.html#plugin-management?

3

Hi @amy.blais,
here is a screenshot from our upgraded Mattermost instance.
Is that what you asked for or do you need to see an extract from our config.json ?
Thanks
Marco

2019-11-18%2015_54_00-System%20Console%20-%20Mattermost
2019-11-18%2015_54_00-System%20Console%20-%20Mattermost927×612 36.9 KB

4

Where does the link to the marketplace take you in the help text under Enable Marketplace when you click it?

5

Hi,
this is the link:
https://mattermost.com/pl/default-mattermost-marketplace.html
Thanks again,
Marco Barbaro
Direzione Sistemi Informativi

PAM Panorama S.P.A.

tel. 041-5495104

  • marco_barbaro@gruppopam.it

Da:
“Amy Blais via
Mattermost Inc.” mattermost@discoursemail.com

Per:
marco_barbaro@gruppopam.it

Data:
19/11/2019 02:19

Oggetto:
[Mattermost
Forum] [Trouble Shooting] Mattermost default plugins not visible after
upgrade

6

Hi Amy, do you have any hints to suggest me to solve the issue?
Was Mattermost 4.5 version too old to upgrade directly to 5.16.3 version? Should I have to upgrade to an intermediate version first?

Thanks again
Best Regards
Marco

7

@mbarbaro Can you help share your config.json settings related to plugins?

8

Sure @amy.blais here it is:

"PluginSettings": {
    "Enable": true,
    "EnableUploads": true,
    "Directory": "./plugins",
    "ClientDirectory": "./client/plugins",
    "Plugins": {},
    "PluginStates": {
        "jira": {
            "Enable": true
        }
    }
 }

Thanks
Marco

9

@mbarbaro, in your logs there’s one item that stands out:

{“level”:“error”,“ts”:1574069260.660455,“caller”:“mlog/log.go:174”,“msg”:“An error occurred saving the configuration”,“path”:"/api/v4/config",“request_id”:“jt3foiw9n7d1d8neixy5cwjpxe”,“ip_addr”:“172.16.20.99”,“user_id”:“1c49g3xy9fr7zx6kjhdd41exmc”,“method”:“PUT”,“err_where”:“saveConfig”,“http_code”:500,“err_details”:“failed to persist: failed to write file: open /data/mattermost/versions/mattermost-4.5.0/config/config.json: permission denied”}

This sounds like the root of the issue. Is the config pointing back at the old 4.5.0 spot, but doesn’t have permission to update it?

10

Hi @jesse , thanks for your hint.
I modified config.json as suggested
and then I notice a new log error as if “client/plugins” dir
was not present even if it was into mattermost dir.
So I read about some threads highlighting
that mattermost was searching for “client” dir into “mattermost/bin”
dir and decided to try to create another “client” dir inside
“bin” and tried to restart mattermost service but it stopped
to work.
Looking at logs it was trying to find
for “root.html” file into “mattermost/bin/client” dir
!
I copied “root.html” from
mattermost/client into mattermost/bin/client and suddenly I was able to
enable plugins into Mattermost!
What do you think about this? Why Mattermost
was looking for “client” dir into “bin” dir in your
opinion?
Thanks again.
Best regards.
Marco Barbaro
Direzione Sistemi Informativi

PAM Panorama S.P.A.

tel. 041-5495104

  • marco_barbaro@gruppopam.it

Da:
“Jesse Hallam
via Mattermost Inc.” mattermost@discoursemail.com

Per:
marco_barbaro@gruppopam.it

Data:
21/11/2019 17:36

Oggetto:
[Mattermost
Forum] [Trouble Shooting] Mattermost default plugins not visible after
upgrade

11

Hi @jesse and @amy.blais is there a way to set Mattermost “working.dir”?
As I supposed, looking deeper at the product logs, the istance working dir starts from “bin”:

{“level”:“info”,“ts”:1574413783.6823297,“caller”:“app/server.go:219”,“msg”:“Current working directory is /data/mattermost/versions/mattermost-4.5.0/bin”}

Perhaps this is the start of the issues I had with plugins.
Could you please help me modifying that setting?
Thanks
Marco

12

Hey @mbarbaro – the working directory should just be the directory from which the mattermost binary is invoke. For example, if you’re in $HOME and you invoke the binary manually, the working directory will match $HOME.

Depending on your operating system and launching mechanism, there are ways of defining that directory before starting mattermost. For example, systemd uses WorkingDirectory as a configuration. Perhaps you can tune that?

1 Like
13

Thanks @jesse for your hint.
At the end I modified WorkingDirectory into systemd mattermost service removing /bin and it worked!
Marco

14

Hi, @mbarbaro

Glad to know that @jesse helped to resolve the case. Just wanted clarification on a couple of things:

Since Jesse mentioned about the config.json still pointing to the old version, does that mean that you had 2 separate directories after the upgrade? For example:

  • /data/mattermost/versions/mattermost-4.5.0
  • /data/mattermost/versions/mattermost-5.16.3

Do you also happen to have the error snippet of this part?

Lastly, can I also confirm if you modified the WorkingDirectory in /lib/systemd/system/mattermost.service to the following to make it work?

WorkingDirectory=/opt/mattermost
15

Hi @ahmaddanial I’ll reply to your questions:

does that mean that you had 2 separate directories after the upgrade?

----> No, I put the new version files into old directory as suggested by mattermost upgrade guide.

Do you also happen to have the error snippet of this part?
Lastly, can I also confirm if you modified the WorkingDirectory in /lib/systemd/system/mattermost.service to the following to make it work?

----> These 2 issues got solved modifying WorkingDirectory into existing 4.5.0 mattermost.service, removing the “/bin” it had at the end of the path.

Thanks
Marco