Bad response from token request after team creation auth

Hi Guys,

I can’t create a team in a gitlab-mattermost environnement. After authorisation from gitlab SSO i got redirected to the /signup/gitlab/complete and results in a 500 error : Bad response from token request.

i don’t use https / certificates, just plain http. I enabled error logging but i don’t have the error details shown. I’am using apache on plesk, and gitlab works perfectly

/var/log/gitlab/mattermost/mattermost.log:

[2016/02/14 20:18:42 CET] [DEBG] /signup/gitlab/complete
[2016/02/14 20:18:42 CET] [EROR] /signup/gitlab/complete:AuthorizeOAuthUser code=500 rid=45puht4ewirytdxzw4kcebw5xa uid= ip=XX.XX.XX.XX Bad response from token request [details: ]

gitlab.rb

mattermost[‘log_enable_file’] = true
mattermost[‘log_file_level’] = ‘DEBUG’
mattermost_external_url ‘http://mattermost.mydomain.com
mattermost_nginx[‘enable’] = false
mattermost_nginx[‘redirect_http_to_https’] = false
mattermost[‘service_use_ssl’] = false

config.json

"LogSettings": {
    "ConsoleEnable": true,
    "ConsoleLevel": "INFO",
    "EnableFile": true,
    "FileLevel": "DEBUG",
    "FileFormat": "",
    "FileLocation": "/var/log/gitlab/mattermost/mattermost.log"
},
"GitLabSettings": {
    "Enable": true,
    "Secret": "mysecret",
    "Id": "myid",
    "Scope": "",
    "AuthEndpoint": "http://gitlab.mydomain.com/oauth/authorize",
    "TokenEndpoint": "http://gitlab.mydomain.com/oauth/token",
    "UserApiEndpoint": "http://gitlab.mydomain.com/api/v3/user"
}

production.log

Started GET
“/oauth/authorize?response_type=code&client_id=e94bcbe68270faa5d259794498c2809132e79e36936f9222474239a7ff724b1e&redirect_uri=http%3A%2F%2Fmattermost.mydomain.com%2Fsignup%2Fgitlab%2Fcomplete&state=eyJhY3Rpb24iOiJzaWdudXAiLCJoYXNoIjoiJDJhJDEwJHg2QmtrRnU5MHkuNlA4ZE02RFUzMy5ONUQvb3FxWjRqT1Q5RnIwOExqdUIuby8xeTlUTW42IiwidGVhbSI6ImFsaWNld29yazAxMjM0NTY3ODkxMDExIn0%3D”
for 78.249.179.124 at 2016-02-14 15:52:10 +0100

Processing by Oauth::AuthorizationsController#new as HTML

Parameters: {“response_type”=>“code”,
“client_id”=>“e94bcbe68270faa5d259794498c2809132e79e36936f9222474239a7ff724b1e”,

“redirect_uri”=>“http://mattermost.mydomain.com/signup/gitlab/complete”,

“state”=>“eyJhY3Rpb24iOiJzaWdudXAiLCJoYXNoIjoiJDJhJDEwJHg2QmtrRnU5MHkuNlA4ZE02RFUzMy5ONUQvb3FxWjRqT1Q5RnIwOExqdUIuby8xeTlUTW42IiwidGVhbSI6ImFsaWNld29yazAxMjM0NTY3ODkxMDExIn0=”}

Completed 200 OK in 71ms (Views: 48.9ms | ActiveRecord: 8.1ms)

Started POST “/oauth/authorize” for 78.249.179.124 at 2016-02-14 15:52:13 +0100

Processing by Oauth::AuthorizationsController#create as HTML

Parameters: {“utf8”=>“✓”, “authenticity_token”=>“[FILTERED]”,

“client_id”=>“e94bcbe68270faa5d259794498c2809132e79e36936f9222474239a7ff724b1e”,

“redirect_uri”=>“http://mattermost.mydomain.com/signup/gitlab/complete”,

“state”=>“eyJhY3Rpb24iOiJzaWdudXAiLCJoYXNoIjoiJDJhJDEwJHg2QmtrRnU5MHkuNlA4ZE02RFUzMy5ONUQvb3FxWjRqT1Q5RnIwOExqdUIuby8xeTlUTW42IiwidGVhbSI6ImFsaWNld29yazAxMjM0NTY3ODkxMDExIn0=”,
“response_type”=>“code”, “scope”=>“api”}

Redirected to
http://mattermost.mydomain.com/signup/gitlab/complete?code=54f138af81298e09d994427365dec4b4e64e49661050f2a35b3d85d04f60d3f6&state=eyJhY3Rpb24iOiJzaWdudXAiLCJoYXNoIjoiJDJhJDEwJHg2QmtrRnU5MHkuNlA4ZE02RFUzMy5ONUQvb3FxWjRqT1Q5RnIwOExqdUIuby8xeTlUTW42IiwidGVhbSI6ImFsaWNld29yazAxMjM0NTY3ODkxMDExIn0%3D

Completed 302 Found in 73ms (ActiveRecord: 56.1ms)

can someone help ?

HI @alicework, from the looks of it your config all seems correct.

Can you see if your GitLab log makes any reference to a POST to http://gitlab.mydomain.com/oauth/token?

It looks like GitLab is returning incorrect data after a POST is made to that URL attempting to exchange the access code for an access token.

Hi @jwilander,

Thanks for taking the time to answer me.

I can see with Firefox developper tools a POST request to http://gitlab.mydomain.com/oauth/authorize resulting in a 302 Found response after authentification on gitlab, but nothing in mattermost log (see below)

utf8=✓
authenticity_token=vt72DvWjnXzLESh+3MGs2XPAOr3UzuI1Yvc+PP+zZzWw6wBElqbHlAExAK3KN4z+eScG+hF2DKXfQ/yyNJgU8Q==
client_id=9d4021472038fffa615fa757550879e39dd7e50dd4c86d72bdb39c7c2ed8f334
redirect_uri=http://mattermost.mydomain.com/signup/gitlab/complete
state=eyJhY3Rpb24iOiJzaWdudXAiLCJoYXNoIjoiJDJhJDEwJHRETlA0SFZqdTh4ZlphVXh4a2pVUU94cms2VEp6OS5yZEQ5Mi5id1gwaHBid1h4ZlEySWwyIiwidGVhbSI6ImFsaWNld29yazAxMjM0NTY3ODkxMDExMTIxMzE0MTUxNjE3MTgxOTIwMjEyMjIzMjQyNTI2MjcifQ==
response_type=code
scope=api

Here’s my mattermost log :

[2016/02/26 14:35:19 CET] [DEBG] /
[2016/02/26 14:35:27 CET] [DEBG] /api/v1/teams/create_with_sso/gitlab
[2016/02/26 14:35:27 CET] [DEBG] /alicework01234567891011121314151617181920212223242526/signup/gitlab
[2016/02/26 14:35:31 CET] [DEBG] /signup/gitlab/complete
[2016/02/26 14:35:31 CET] [EROR] /signup/gitlab/complete:AuthorizeOAuthUser code=500 rid=4b6e4jy85jyctkrezkx51sckqo uid= ip=78.249.179.124 Bad response from token request [details: ]

Ok, it certainly looks the first step of the authentication process is working correctly. There are three steps that must be completed for the GitLab SSO to work fully:

  1. Receive an access code from http://gitlab.mydomain.com/oauth/authorize (which looks to be happening correctly)
  2. Exchange that access code for an access token through http://gitlab.mydomain.com/oauth/token (where we seem to be failing)
  3. Retrieve the user object from http://gitlab.mydomain.com/api/v3/user

If these work, they should all happen quickly and you as the user should hardly notice anything happening at all.

If you look in the server log for GitLab (the production.log), do you see a request being made to http://gitlab.mydomain.com/oauth/token? How GitLab responds to that request is causing the failure on the second step and if the log shows anything it could be very useful in figuring out the issue.

1 Like

Nothing like a request to /oauth/token in my production log (just emptied it before trying to create a team). The whole process is something like :

Started GET "/oauth/authorize
...
Started POST "/oauth/authorize"
...
Redirected to http://mattermost.mydomain.com/signup/gitlab/complete
Completed 302 Found in 71ms (ActiveRecord: 46.8ms)

The only reference to a token I can found is the “authenticity_token” parameter of the POST /oauth/authorize :

Processing by Oauth::AuthorizationsController#create as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", ....

Hmmm, that’s interesting. It’s acting as if it’s not hitting the GitLab url at all but Mattermost is hitting a code path where it must have received some successful (though, incorrect) response from that URL.

Can you double check that your token URL is correct?

Can you also try setting “EnableInsecureOutgoingConnections” to true in your config.json (it’s under Service Settings)?

Thanks again for the help,

I double checked the URLs in config and all is good, but if i go to http://gitlab.XXXXXXXXXXX.com/oauth/token in my browser there is indeed a 404 not found response from gitlab, whereas the other 2 urls (/authorize and /api/V3/user) works (401 or gitlab error message)

"GitLabSettings": {
        "Enable": true,
        "Secret": "mysecret",
        "Id": "myid",
        "Scope": "",
        "AuthEndpoint": "http://gitlab.XXXXXXXXXXX.com/oauth/authorize",
        "TokenEndpoint": "http://gitlab.XXXXXXXXXXX.com/oauth/token",
        "UserApiEndpoint": "http://gitlab.XXXXXXXXXXX.com/api/v3/user"
    }

I didn’t find an “EnableInsecureOutgoingConnections” entry in my mattermost config.json file, here is the “ServiceSettings” section :

"ServiceSettings": {
        "ListenAddress": "127.0.0.1:8065",
        "MaximumLoginAttempts": 10,
        "SegmentDeveloperKey": "",
        "GoogleDeveloperKey": "",
        "EnableOAuthServiceProvider": false,
        "EnableIncomingWebhooks": false,
        "EnableOutgoingWebhooks": false,
        "EnablePostUsernameOverride": false,
        "EnablePostIconOverride": false,
        "EnableTesting": false,
        "EnableSecurityFixAlert": true,
        "EnableDeveloper": false,
        "SessionLengthWebInDays" : 30,
        "SessionLengthMobileInDays" : 30,
        "SessionLengthSSOInDays" : 30,
        "SessionCacheInMinutes" : 10
    }, 

I added "EnableInsecureOutgoingConnections" : true but it doesn’t seems to change anything

do you think i should start an issue in the gitlab-mattermost deposit ?

Sorry for the late response, we’re going through a release that’s taking up a bunch of my time. Opening an issue with GitLab is probably a good idea. I’ve tried to recreate your issue with a local setup of GitLab (latest version) but it seems to be working fine for me.

Which version of Mattermost and GitLab are you using? If it’s not latest maybe try updating to latest to see if that helps

Ok, i’am gonna try updating gitlab to latest 8.5. Should i update mattermost to v2 too ?

Mattermost
1.4.0

GitLab
8.4.4

GitLab Shell
2.6.10

GitLab API
v3

Git
2.6.2

Ruby
2.1.8p440

Rails
4.2.5.1

Yes, try updating to Mattermost 2.0 as well

Well, I updated gitlab and mattermost to latest 8.5 / 2.0, but i still can’t create a team. Nothing changed, same error and nothing on logs.

Tried EnableInsecureOutgoingConnections : true too.

Gonna raise an issue in the repo and hope for some solution there.

Thanks again for the help jwilander

Edit :

Posting my apache conf (from my vhost on plesk), in case someone sees something wrong :

mattermost.mydomain.com

ServerName mattermost.mydomain.com
ServerAlias mattermost.mydomain.com

#DocumentRoot /opt/gitlab/embedded/service/mattermost/web

ProxyPreserveHost On
RewriteEngine On

RewriteCond %{REQUEST_URI}  ^/api/v1/websocket    [NC,OR]
RewriteCond %{HTTP:UPGRADE} ^WebSocket$           [NC,OR]
RewriteCond %{HTTP:CONNECTION} ^Upgrade$          [NC]
RewriteRule .* ws://127.0.0.1:8065%{REQUEST_URI}  [P,QSA,L]

RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule .* http://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]

# Be sure to uncomment the next 2 lines if https is used
# RequestHeader set X-Forwarded-Proto "https"
# Header set Strict-Transport-Security "max-age=31536000; includeSubDomains"

# Prevent apache from sending incorrect 304 status updates
RequestHeader unset If-Modified-Since
RequestHeader unset If-None-Match

<Location /api/v1/websocket>
    Require all granted
    ProxyPassReverse ws://127.0.0.1:8065/api/vi/websocket
    ProxyPassReverseCookieDomain 127.0.0.1 mattermost.mydomain.com
</Location>

<Location />
    Require all granted
    ProxyPassReverse http://127.0.0.1:8065/
    ProxyPassReverseCookieDomain 127.0.0.1 mattermost.mydomain.com
</Location>

gitlab.mydomain.com

ServerName gitlab.mydomain.com
ServerSignature Off

ProxyPreserveHost On

# Ensure that encoded slashes are not decoded but left in their encoded state.
# http://doc.gitlab.com/ce/api/projects.html#get-single-project
AllowEncodedSlashes NoDecode

<Location />
    # New authorization commands for apache 2.4 and up
    # http://httpd.apache.org/docs/2.4/upgrading.html#access
    Require all granted

    #Allow forwarding to gitlab-workhorse
    ProxyPassReverse http://127.0.0.1:8181
    ProxyPassReverse http://gitlab.mydomain.com/
</Location>

# Apache equivalent of nginx try files
# http://serverfault.com/questions/290784/what-is-apaches-equivalent-of-nginxs-try-files
# http://stackoverflow.com/questions/10954516/apache2-proxypass-for-rails-app-gitlab
RewriteEngine on

#Forward these requests to gitlab-workhorse
RewriteRule .* http://127.0.0.1:8181%{REQUEST_URI} [P,QSA]

# needed for downloading attachments
DocumentRoot /opt/gitlab/embedded/service/gitlab-rails/public

#Set up apache error documents, if back end goes down (i.e. 503 error) then a maintenance/deploy page is thrown up.
ErrorDocument 404 /404.html
ErrorDocument 422 /422.html
ErrorDocument 500 /500.html
ErrorDocument 503 /deploy.html

# It is assumed that the log directory is in /var/log/httpd.
# For Debian distributions you might want to change this to
# /var/log/apache2.
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common_forwarded
ErrorLog /var/log/httpd/gitlab.mydomain.com_error.log
CustomLog /var/log/httpd/gitlab.mydomain.com_forwarded.log common_forwarded
CustomLog /var/log/httpd/gitlab.mydomain.com_access.log combined env=!dontlog
CustomLog /var/log/httpd/gitlab.mydomain.com.log combined

Exactly the same problem, and the same logging. We both must be doing the same things wrong!

Aaaannnnd, I fixed it! Turns out I forgot to correctly set the ‘gitlab_auth_endpoint’, ‘gitlab_token_endpoint’, and the ‘gitlab_user_api_endpoint’ settings.

Good for you !

I’m still stuck :confused:

would you mind copying me your gitlab config ? (without your domain names)

Sure, here it is. I replaced my domain with mydomain.com.
The changes I made are actually very minimal.

Thanks for sharing

My config is similar for mattermost. I can’t understand why it doesn’t work :confused:

Same problem here. Did you find a solution ?

Same problem for me. Any solution?

Thanks!

I also experienced this problem when switching my HTTP Gitlab instance over to HTTPS. I had to update my /etc/gitlab.gitlab-secrets.json to use HTTPS for the *_endpoint parameters. I don’t think Mattermost will follow the 302 on the token request if you’re redirecting HTTP to HTTPS. (and rightfully so)