GitLab Mattermost Token request failed

Troubleshooting
1

I have both gitlab and mattermost running with https using Let’s Encrypt certificates. I added the matter most application with the following callback URLs

https://mattermost.mysite.com/signup/gitlab/complete
https://mattermost.mysite.com/login/gitlab/complete

I then edited my /etc/gitlab/gitlab.rb by adding the following:

external_url "https://gitlab.mysite.com"
nginx['redirect_http_to_https'] = true
nginx['ssl_certificate'] = "/etc/letsencrypt/live/gitlab.mysite.com/cert.pem"
nginx['ssl_certificate_key'] = "/etc/letsencrypt/live/gitlab.mysite.com/privkey.pem"

mattermost_external_url 'https://mattermost.mysite.com'
mattermost_nginx['redirect_http_to_https'] = true
mattermost_nginx['ssl_certificate'] = "/etc/letsencrypt/live/mattermost.mysite.com/0000_cert.pem"
mattermost_nginx['ssl_certificate_key'] = "/etc/letsencrypt/live/mattermost.mysite.com/privkey1.pem"
mattermost['service_use_ssl'] = true

mattermost['gitlab_enable'] = true
mattermost['gitlab_secret'] = "pooppooppooppooppoop"
mattermost['gitlab_id'] = "peepeepeepeepee"
mattermost['gitlab_scope'] = ""
mattermost['gitlab_auth_endpoint'] = "https://gitlab.mysite.com/oauth/authorize"
mattermost['gitlab_token_endpoint'] = "https://gitlab.mysite.com/oauth/token"
mattermost['gitlab_user_api_endpoint'] = "https://gitlab.mysite.com/api/v3/user"

Then When I go to https://mattermost.mysite.com I see

Mattermost home

followed by this after entering a new team name and clicking Create Team with GitLab account

Mattermost authorize

and finally I get the error

Mattermost error

In /var/log/gitlab/mattermost/mattermost.log I see this error

[2016/01/28 21:15:48 EST] [EROR] /signup/gitlab/complete:AuthorizeOAuthUser code=500 rid=5j3eqjfrj7gpoopsxk6c9h8ka uid= ip=10.34.6.4 Token request failed [details: Post https://gitlab.mysite.com/oauth/token: x509: certificate signed by unknown authority]

Any ideas on how to fix this? Is Let’s Encrypt an unknown authority? Is there a way to get to know Let’s Encrypt?

I’m running GitLab Community Edition 8.4.2 8d07b81 on Debian GNU/Linux 7.8 (wheezy)

Thanks

2

Please see http://docs.mattermost.com/install/troubleshooting.html#x509-certificate-signed-by-unknown-authority

3

Including some addtional information that I found userful.

I have this same issue, but wanted to document how I solved this issue since this is one of the top google search results regarding the x509: certificate signed by unknown authority issue.

I read through all the other links / issues mentioned here and did run across a couple others

But ultimately, I would recommend reading the following blog post that describes why SSL certs are different on different systems and why this causes so much headache:

In my specific use case, we run a RHEL / Centos 6 server.

While I had already added my Internal CA root certificate to the /etc/pki/ca-trust/source/anchors/ directory and then ran the update-ca-trust as root. This updates certificates in the /etc/pki/ca-trust/extracted directory.

After updating this and forcing the GitLab CA certificate to point to this extracted certificate, I still could not get Mattermost to connect properly:

[root@server /]# ls -la /opt/gitlab/embedded/ssl/certs/
total 8
drwxr-xr-x. 2 root root 4096 Jul 21 13:19 .
drwxr-xr-x. 4 root root 4096 Jul 21 12:47 ..
lrwxrwxrwx. 1 root root   55 Jul 21 13:02 cacert.pem -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt

It wasn’t until I saw that there was another certs directory from the HappyAssassin blog post that was not being updated. The /etc/ssl/certs is a sym link to /etc/pki/tls/certs and in here these certificates did not have my internal root CA certificate.

So I removed these and pointed this these to use certificates from the extracted directory:

[root@server /]# ls -la /etc/ssl/
total 16
drwxr-xr-x.   2 root root  4096 Jul 21 08:30 .
drwxr-xr-x. 122 root root 12288 Jul 21 13:29 ..
lrwxrwxrwx.   1 root root    16 Jul 21 08:30 certs -> ../pki/tls/cert
[root@server /]# ls -la /etc/ssl/certs/
total 20
drwxr-xr-x. 2 root root 4096 Jul 21 13:27 .
drwxr-xr-x. 5 root root 4096 Jul 21 08:30 ..
lrwxrwxrwx. 1 root root   49 Jul 21 13:27 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx. 1 root root   55 Jul 21 13:27 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
-rwxr-xr-x. 1 root root  610 May  9 08:32 make-dummy-cert
-rw-r--r--. 1 root root 2242 May  9 08:32 Makefile
-rwxr-xr-x. 1 root root  829 May  9 08:32 renew-dummy-cert

It wasn’t until I updated these then did one more gitlab reconfigure that Mattermost would finally recognize the certificates from GitLab.

I’m not sure if this is something that is encoded into Mattermost or possibly one of its dependencies, but this was pretty difficult to track down especially since every system does SSL certificates differently and applications/software all look for these in different ways.

Hopefully this helps someone else or at least provides them with background on the SSL issues since that blog post really helped me figure out the core issue here.

4

I have this exact issue.
gitlab & mattermost running together (in my case in the same docker container).
As per gitlab documentation, it is enough to put the certificate under:
/etc/gitlab/trusted-certs/my_full_chain_ca.crt

After reconfigure, I do find it linked under: /opt/gitlab/embedded/ssl/certs/ but mattermost is still complaining with:
[2016/10/22 22:34:58 IDT] [EROR] /signup/gitlab/complete:AuthorizeOAuthUser code=500 rid=dddddddddddduid= ip=10.x.x.x Token request failed [details: Post https://gitlab.aaaaa.com/oauth/token: x509: certificate signed by unknown authority]

I checked, and my_full_chain_ca.crt is not linked automatically under: /etc/ssl/certs. Should it be there? If so, shouldn’t reconfigure take care of this as well?

5

Hi @ReSearchITEng,

Do you have SSL client certificates enabled on your GitLab instance? If so, try turning that off and trying again

6

Hi @jwilander,
Thanks for the answer. Yes, it’s off, as in the default config. I am using 8.13.0 (if it matters).

[root@gitlab config]# grep ssl_client gitlab.rb
# nginx['ssl_client_certificate'] = "/etc/gitlab/ssl/ca.crt" # Most root CA's are included by default
7

OK, @ReSearchITEng can you try setting the following in your GitLab config?

mattermost['service_enable_insecure_outgoing_connections'] = true

That should allow Mattermost to connect to servers that are using self-signed SSL certificates.

1 Like