System console not appearing for admin user

Troubleshooting
1

After a few config changes in a bitnami AWS Mattermost instance, the two users with admin access no longer see the system console in their hamburger menu

Following steps found in a search in this forum, I found the mattermost/server/bin directory and ran

sudo ./mattermost roles system_admin admin

(“admin” is the username for the default user with admin access)

I get the following output:

{“level”:“info”,“ts”:1543243922.0535872,“caller”:“utils/i18n.go:78”,“msg”:“Loaded system translations for ‘en’ from ‘/opt/bitnami/apps/mattermost/server/i18n/en.json’”}
{“level”:“info”,“ts”:1543243922.053817,“caller”:“app/app.go:204”,“msg”:“Server is initializing…”}
{“level”:“info”,“ts”:1543243922.0587118,“caller”:“sqlstore/supplier.go:209”,“msg”:“Pinging SQL master database”}
{“level”:“info”,“ts”:1543243922.0767624,“caller”:“sqlstore/channel_store_experimental.go:47”,“msg”:“Enabling experimental public channels materialization”}
{“level”:“info”,“ts”:1543243922.1274724,“caller”:“app/plugin.go:100”,“msg”:“Starting up plugins”}
{“level”:“error”,“ts”:1543243922.1276336,“caller”:“app/plugin.go:108”,“msg”:“Failed to start up plugins”,“error”:“mkdir ./client/plugins: no such file or directory”}
{“level”:“info”,“ts”:1543243922.136283,“caller”:“app/app.go:283”,“msg”:“Stopping Server…”}
{“level”:“info”,“ts”:1543243922.136381,“caller”:“app/web_hub.go:120”,“msg”:“stopping websocket hub connections”}
{“level”:“warn”,“ts”:1543243922.1364763,“caller”:“app/web_hub.go:125”,“msg”:“We appear to have already sent the stop checking for deadlocks command”}
{“level”:“info”,“ts”:1543243922.1376143,“caller”:“sqlstore/supplier.go:852”,“msg”:“Closing SqlStore”}
{“level”:“info”,“ts”:1543243922.1380422,“caller”:“app/app.go:305”,“msg”:“Server stopped”}

2

I should specify, the config changes I’m referring to that happened before the admin console disappeared involve changing IP addresses, adding a DNS name and adding an SSL certificate.

3

Update: I created a …bin/client/plugins directory, which seems a weird place to put it, but whatever, so that one error message disappears

The output is now:
{“level”:“info”,“ts”:1543246021.4875233,“caller”:“utils/i18n.go:78”,“msg”:“Loaded system translations for ‘en’ from ‘/opt/bitnami/apps/mattermost/server/i18n/en.json’”}
{“level”:“info”,“ts”:1543246021.487746,“caller”:“app/app.go:204”,“msg”:“Server is initializing…”}
{“level”:“info”,“ts”:1543246021.4892926,“caller”:“sqlstore/supplier.go:209”,“msg”:“Pinging SQL master database”}
{“level”:“info”,“ts”:1543246021.5083587,“caller”:“sqlstore/channel_store_experimental.go:47”,“msg”:“Enabling experimental public channels materialization”}
{“level”:“info”,“ts”:1543246021.5695505,“caller”:“app/plugin.go:100”,“msg”:“Starting up plugins”}
{“level”:“info”,“ts”:1543246022.1409373,“caller”:“app/app.go:283”,“msg”:“Stopping Server…”}
{“level”:“info”,“ts”:1543246022.1409864,“caller”:“app/web_hub.go:120”,“msg”:“stopping websocket hub connections”}
{“level”:“warn”,“ts”:1543246022.1409993,“caller”:“app/web_hub.go:125”,“msg”:“We appear to have already sent the stop checking for deadlocks command”}
{“level”:“info”,“ts”:1543246022.1412697,“caller”:“app/plugin.go:156”,“msg”:“Shutting down plugins”}
{“level”:“info”,“ts”:1543246022.141965,“caller”:“sqlstore/supplier.go:852”,“msg”:“Closing SqlStore”}
{“level”:“info”,“ts”:1543246022.142279,“caller”:“app/app.go:305”,“msg”:“Server stopped”}

I restart everything, including mysql and nginx, and still no admin console access

When I go to https://mysiteurl/admin_console I get a blank page

4

Hi @Prooffreader,

Is this setting enabled on your server: https://docs.mattermost.com/administration/config-settings.html#enable-cross-origin-requests-from?

5

Hi @amy.blais, thanks for responding!

I had “AllowCorsFrom” set to “”; I just changed it to “*” (I have “SiteURL” set to my domain name) and restarted everything, and the problem still exists, nothing seems to have changed.

6

Interesting, the corporate reverse proxy I’m using to access the site is screwing with the headers and telling me there are websocket problems, accessing it on my laptop with my phone as a hotspot shows no problems, I can see the admin console and everything.

The joys of working for a multinational company with tight infosec rules.

1 Like
7

I had exactly the same issue. I ran Mattermost 3.9.1 on a Bitnami stack on a private cloud (not AWS) and just upgraded to 5.6.2.

I cannot reach the system console from the admin account. When I try to assign again the admin role to the admin account, it generates exactly the same messages.

To be noted as well, I get a websocket error (" Please check connection, Mattermost unreachable. If issue persists, ask administrator to [check WebSocket port"). All my attempts to adapt the config files according to the Mattermost (and less voluminous Bitnami) guidance did not solve it.

Any help would be most welcome. I must solve this issue in 36hours or I will have to revert to my previous Mattermost version. If you had any clue on this, I would be more than grateful.

Thanks !

8

Hey @alexbab - can you try these troubleshooting steps to start? https://docs.mattermost.com/install/troubleshooting.html#please-check-connection-mattermost-unreachable-if-issue-persists-ask-administrator-to-check-websocket-port

9

Hi @jasonblais, thank you. Yes I have already looked at this page and the related links.
The Bitnami install includes already the SSL code. The only changes I made is in

  • location ~ /api/v[0-9]+/(users/)?websocket$ which solved some of the mistakes (the previous MM was on API v3). Looking at the mattermost.log, it seems however that there is somewhere
  • I assigned “*” to “AllowCorsFrom”.
  • And for proxy_set_header Host $host;, it is in the mattermost.conf file proxy_set_header Host $http_host;

In general, I have been browsing the Mattermost and Bitnami forum for the issue on the system console and on the websocket (which may be linked somehow?). There are posts but nothing is working.

in the mattermost.log file, the error level messages are:

  • Failed to start up plugins* ",“error”:“mkdir ./plugins: permission denied”} I don’t know here which permissions to assign (I already assigned chown to mattermost for server)
  • websocket connect err: websocket: the client is not using the websocket protocol: ‘upgrade’ token not found in ‘Connection’ header"
  • Failed to upgrade websocket connection
  • Invalid or missing channel_id parameter in request URL

Did you have any other hint?

10

Thank you @alexbab, this is helpful.

Given you upgraded from a relatively old version (3.9), I recommend reviewing our important upgrade notes here: https://docs.mattermost.com/administration/important-upgrade-notes.html

Make sure to review and take actions on those that apply to you. One additional thing that comes to mind is setting the Site URL field in System Console > Configuration, if you haven’t already done so.

11

Hi, try open your instance in web browser “IE”

12

I don’t know if this will help your case or not, but when I try to use the MatterMost CLI, I usually have sudo use the mattermost user. For example:

sudo -u mattermost <path to mattermost binary>/mattermost team add example_team example_user

13

Also, you may wish to look inside the database that you are using. In the Users table, there is a Roles column, and for admins, the Roles look to have system_admin system_user. Regular users just seem to have only system_user under Roles. It may be possible to update the value in the database manually for admin access.

14

Note that this heading in the troubleshooting guide corresponds to your error message: https://docs.mattermost.com/install/troubleshooting.html#failed-to-upgrade-websocket-connection

I got the same error message after proxying mattermost through nginx; I solved this by forwarding WebSocket headers like this:

server {
    location / {
        proxy_pass http://localhost:8065;
        # following three lines added to forward WebSocket headers
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";