Unable to access system console, permission error in log


Not sure which update this happened with but I’ve noticed I can no longer access system console. The link is in the menu but only a blank page is displayed.

I notice a request is made to /api/v3/admin/config which is met with a 403 error and the following (which is also logged in the main log.):

{"id":"api.context.permissions.app_error","message":"You do not have the appropriate permissions","detailed_error" :"","request_id":"tyf9a86ke3ys9qqkai3kfkwsnr","status_code":403}

The url it is trying to access is: /admin_console/system_analytics

The user has the roles: “admin system_admin”

Any ideas?


P.S. This was installed with gitlab omnibus.

Anybody? … I’m not sure what to do next.

Hey @traxwriter,

Wondering if something went wrong with permissions on the account.

Can you try to make another account System Admin and see if the issue reproduces? You can use the platform roles system_admin CLI command to promote somone to a System Admin.

You can also remove system admin privileges from a user first (using platform roles member CLI), then promote them back to a System Admin.

Let me know if the above doesn’t help resolve the issue.

Hi @jasonblais

Sorry for the delay, been really busy here.

I’ve tried switching roles around but I’m getting a panic which I can’t find a solution for.

When running sudo -u mattermost /opt/gitlab/embedded/bin/mattermost roles system_admin traxwriter -c /var/opt/gitlab/mattermost/config.json

I get
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x418d58]

goroutine 1 [running]: panic(0xc72560, 0xc4200100a0) /usr/local/go/src/runtime/panic.go:500 +0x1a1 main.getUserFromUserArg(0x7ffda1a88753, 0xb, 0x1) /home/ubuntu/workspace/msr/platform-release/src/github.com/mattermost/platform/cmd/platform/userargs.go:21 +0x28 main.getUsersFromUserArgs(0xc4201156e0, 0x1, 0x3, 0xc4210c3ba8, 0x9b41b6, 0xc30d60) /home/ubuntu/workspace/msr/platform-release/src/github.com/mattermost/platform/cmd/platform/userargs.go:13 +0xa2 main.makeSystemAdminCmdF(0x11466c0, 0xc4201156e0, 0x1, 0x3, 0x0, 0x0) /home/ubuntu/workspace/msr/platform-release/src/github.com/mattermost/platform/cmd/platform/roles.go:46 +0x75 github.com/mattermost/platform/vendor/github.com/spf13/cobra.(*Command).execute(0x11466c0, 0xc420115140, 0x3, 0x3, 0x11466c0, 0xc420115140) /home/ubuntu/workspace/msr/platform-release/src/github.com/mattermost/platform/vendor/github.com/spf13/cobra/command.go:643 +0x23e github.com/mattermost/platform/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc42113e240, 0x0, 0x0, 0x0) /home/ubuntu/workspace/msr/platform-release/src/github.com/mattermost/platform/vendor/github.com/spf13/cobra/command.go:734 +0x367 github.com/mattermost/platform/vendor/github.com/spf13/cobra.(*Command).Execute(0xc42113e240, 0xc420106670, 0xb) /home/ubuntu/workspace/msr/platform-release/src/github.com/mattermost/platform/vendor/github.com/spf13/cobra/command.go:692 +0x2b main.main() /home/ubuntu/workspace/msr/platform-release/src/github.com/mattermost/platform/cmd/platform/mattermost.go:50 +0x49f

If I run the help or version commands there is no error only when I ask it to actually do something…

Thanks @traxwriter for the response,

  1. What database (and database version) are you using?
  2. What version of Mattermost (and GitLab) are you on?
  3. If you sign into your Mattermost instance with the account, do you see “System Console” as one of the Mattermost main menu options (from three-dotted menu)? To confirm the role was successfully switched.

You must run the command from inside the mattermost directory. So for you the commands would be something like this:

cd /opt/gitlab/embedded/bin
sudo su mattermost ./mattermost roles system_admin traxwriter -c /var/opt/gitlab/mattermost/config.json


PostgreSQL 9.6.1
Mattermost Version: 3.7.0
Mattermost Build Number: 3.7.2
Gitlab Version: 9.0.2

The system console always shows for my user no matter which command is run, however it cannot be accessed.

Thanks @JeffSchering

If I run the command as root or with sudo I receive permission denied:

[root@git bin]# cd /opt/gitlab/embedded/bin
[root@git bin]# sudo su mattermost ./mattermost roles system_admin traxwriter -c /var/opt/gitlab/mattermost/config.json
./mattermost: /var/opt/gitlab/mattermost/config.json: Permission denied

if I first su to mattermost I get the following:

[root@git bin]# su mattermost
sh-4.2$ ./mattermost roles system_admin traxwriter -c /var/opt/gitlab/mattermost/config.json
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x418d58]

goroutine 1 [running]:
panic(0xc72560, 0xc420012090)
	/usr/local/go/src/runtime/panic.go:500 +0x1a1
main.getUserFromUserArg(0x7ffdbdb946fd, 0xa, 0x1)
	/home/ubuntu/workspace/msr/platform-release/src/github.com/mattermost/platform/cmd/platform/userargs.go:21 +0x28
main.getUsersFromUserArgs(0xc420114fc0, 0x1, 0x3, 0xc420f1dba8, 0x9b41b6, 0xc30d60)
	/home/ubuntu/workspace/msr/platform-release/src/github.com/mattermost/platform/cmd/platform/userargs.go:13 +0xa2
main.makeSystemAdminCmdF(0x11466c0, 0xc420114fc0, 0x1, 0x3, 0x0, 0x0)
	/home/ubuntu/workspace/msr/platform-release/src/github.com/mattermost/platform/cmd/platform/roles.go:46 +0x75
github.com/mattermost/platform/vendor/github.com/spf13/cobra.(*Command).execute(0x11466c0, 0xc420114f30, 0x3, 0x3, 0x11466c0, 0xc420114f30)
	/home/ubuntu/workspace/msr/platform-release/src/github.com/mattermost/platform/vendor/github.com/spf13/cobra/command.go:643 +0x23e
github.com/mattermost/platform/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc42008a240, 0x0, 0x0, 0x0)
	/home/ubuntu/workspace/msr/platform-release/src/github.com/mattermost/platform/vendor/github.com/spf13/cobra/command.go:734 +0x367
github.com/mattermost/platform/vendor/github.com/spf13/cobra.(*Command).Execute(0xc42008a240, 0xc42010e430, 0xb)
	/home/ubuntu/workspace/msr/platform-release/src/github.com/mattermost/platform/vendor/github.com/spf13/cobra/command.go:692 +0x2b
	/home/ubuntu/workspace/msr/platform-release/src/github.com/mattermost/platform/cmd/platform/mattermost.go:50 +0x49f

Just paid more attention to the output, and the paths are weird, where does it get them from?

/hone/ubuntu/ doesn’t exist, neither does /usr/local/go/

small fix, should be sudo -u

Well spotted @prixone although it gives the same output as su to mattermost.

if that gives you permission denied running as root then you are probably missing something there…

Would also check your folders permissions and users with ls -la.

if that is a ubuntu I would go full root instead:

sudo -s
cd /opt/gitlab/embedded/bin
./mattermost roles system_admin traxwriter -c /var/opt/gitlab/mattermost/config.json

You also need to ensure that the mattermost folder and config.json folder are right.

Unhappily I don’t know how gitlab does its thing so I can’t advise further.

@prixone It’s actually running on CentOS 7 hence why it’s a bit weird that it’s trying to use /home/ubuntu

The config file is correct and looks ok. However none of the paths in the panic exist.

Do we know where those paths are defined?

I think that is the dump for debugging which shows paths from where it was compiled at, that is if you’re running the compiled version and not compiling and running from the source.

The path to the bin/mattermost is right as well?

Looks ok to me:

[root@git bin]# ls -la /opt/gitlab/embedded/bin/mattermost 
-rwxr-xr-x 1 root root 22296813 Mar 29 21:14 /opt/gitlab/embedded/bin/mattermost

Are you doing it from source or what file u installed from the downloads?

Or better why don’t you come over at https://pre-release.mattermost.com/ and send me a DM we can talk better there

The install is via gitlab omnibus.

It turns out that you need to be in the mattermost directory only for non GitLab Omnibus installations.

For GitLab omnibus, the format of the command is as follows:

sudo -u mattermost /opt/gitlab/embedded/bin/mattermost --config=/var/opt/gitlab/mattermost/config.json help

It looks like the CLI command that you want to run must be at the end, after the --config option, not before.

Try this command from your home directory to see if it works:

sudo -u mattermost /opt/gitlab/embedded/bin/mattermost  --config=/var/opt/gitlab/mattermost/config.json roles system_admin traxwriter