Mattermost auth failure with GitLab omnibus

Troubleshooting
#1

Summary

When migrating a mattermost install to a new server, using GitLab omnibus, auth fails with ‘Bad response from token request’. I’ve attempted fresh installs multiple times purging everything and trying again multiple ways and the auth always fails.

Related GitLab bug where i’ve had no response yet.

Steps to reproduce

  • Install gitlab-ce omnibus package on Ubuntu 18.04

  • Restore gitlab.rb.

  • Restore gitlab from backup

  • Install mattermost:
    Start gitlab and reconfigure.

    gitlab-ctl start
gitlab-ctl reconfigure
gitlab-ctl restart

  • Fix auth urls in /admin/applications

Here is my entire gitlab.rb (domain redacted), everything else is default:

external_url 'https://git.example.com'

gitlab_rails['gitlab_email_from'] = 'git@example.com'
gitlab_rails['gitlab_email_display_name'] = 'example.com git'
gitlab_rails['gitlab_username_changing_enabled'] = false
gitlab_rails['gitlab_default_projects_features_issues'] = false
gitlab_rails['gitlab_default_projects_features_wiki'] = false
gitlab_rails['backup_path'] = "/var/data/backup"
gitlab_rails['backup_keep_time'] = 691200
git_data_dirs({ 
  "default" => { 
    "path" => "/var/data/git-data"
  } 
})
gitlab_rails['gitlab_shell_ssh_port'] = 27271

nginx['redirect_http_to_https'] = true
nginx['ssl_protocols'] = "TLSv1.2"
nginx['custom_nginx_config'] = "include /etc/gitlab/nginx/sites-enabled/*;"

mattermost_external_url 'https://chat.example.com'
mattermost['enable'] = true
mattermost['team_site_name'] = "example.com Chat"
mattermost['gitlab_enable'] = true
mattermost['gitlab_id'] = "ID"
mattermost['gitlab_secret'] = "SECRET"
mattermost['gitlab_auth_endpoint'] = "https://git.example.com/oauth/authorize"
mattermost['gitlab_token_endpoint'] = "https://git.example.com/oauth/token"
mattermost['gitlab_user_api_endpoint'] = "https://git.example.com/api/v4/user"
mattermost['file_directory'] = "/var/data/mattermost/data"
mattermost_nginx['redirect_http_to_https'] = true
mattermost_nginx['listen_addresses'] = ['*', '[::]']

letsencrypt['enable'] = true

What is the current bug behavior?

Attempt to login - ‘Error Bad response from token request’

I was successful enabling login by email and logging in. Mattermost is working, and the correct urls show up in the system console. I also checked the token in gitlab and the console and they are the same.

#2

Hi @johnramsden,

Wondering if any of these older threads might help:

If not, please let me know and I will ask a developer for more details on this.

#3

@hmhealey Would you know if this is an issue on Mattermost side? The reporter originally posted this issue in Gitlab: https://gitlab.com/gitlab-org/gitlab-ce/issues/60171.

#4

That configuration looks correct to me. If you run gitlab-ctl tail mattermost to view the logs while trying to log in, do those provide a more detailed error message?

#5

I removed the install, but /var/log/gitlab/mattermost/mattermost.log contains:

{"level":"error","ts":1554618871.9515939,"caller":"api4/oauth.go:493","msg":"AuthorizeOAuthUser:
Bad response from token request, response_body="}
{"level":"error","ts":1554618886.7511368,"caller":"web/context.go:52","msg":"Please sign in usin
g gitlab","path":"/api/v4/users/login","request_id":"h11xc1xf4by4tfttpri84ngkxo","ip_addr":"207.
216.83.234","user_id":"","method":"POST","err_where":"login","http_code":400,"err_details":""}

I’ll re-install so I can answer further questions.

#6

I guess that doesn’t provide much more information. I would’ve hoped that the response_body= would’ve included some more information after that instead of an empty string. Perhaps GitLab returned an error response with no body?

Did you happen to find out anything new after re-installing?