Configure stand-alone mattermost with gitlab sign-on

Troubleshooting
1

Hello,
I have installed mattermost as a separate server on its own virtual machine on docker on Ubuntu 18.04. I’m running nginx 1.17 alpine, mattermost database (postgresql) and the mattermost application itself.
On another virtual machine I have gitlab installed. The point is the configure SSO with gitlab, so that I can use LDAP with mattermost Team Edition. I haven’t been able to do so, though, because I keep getting this error:

{"level":"error","ts":1564185931.9620516,"caller":"api4/oauth.go:493","msg":"AuthorizeOAuthUser: Token request failed, Post https://gitlab.example.com/oauth/token: dial tcp: i/o timeout"}

I’m configured gitlab with https, I’ve added a new application in gitlab, copied all the necessary information in the Gitlab section under System Control in mattermost, but I still get this.
Under special circumstances which I haven’t really been able to determine, it sometimes does work after refreshing the mattermost page, despite the “Token request failed” error.
I think it might have worked when I configured gitlab with http only. But the error was still there nonetheless, so the problem sticks, so I doubt it’s actually related to TLS.

Any ideas what is happening? The gitlab server is reachable, I can ping, curl etc., but the POST request seems problematic.
This is my gitlab configuration:

external_url 'https://repos.example.com'
gitlab_rails['ldap_enabled'] = true
gitlab_rails['ldap_servers'] = YAML.load <<-EOS
main: # 'main' is the GitLab 'provider ID' of this LDAP server
     label: 'LDAP'
     host: 'atene-dc01.atene.local'
     port: 636
     uid: 'sAMAccountName'
     bind_dn: 'cn=gitlab-user,ou=Company-Dienstkonten,dc=domaincontroller,dc=local'
     password: '5}#HSZ,7kq-DQI4Ms23?'
     encryption: 'simple_tls' # "start_tls" or "simple_tls" or "plain"
     verify_certificates: true
     active_directory: true
     lowercase_usernames: true
     base: 'ou=Aktive User,ou=atene,dc=domaincontroller,dc=local'
     user_filter: ''
EOS
unicorn['enable'] = true
unicorn['worker_timeout'] = 60
unicorn['worker_processes'] = 2
nginx['redirect_http_to_https'] = true
nginx['ssl_certificate'] = "/etc/gitlab/ssl/#{node['fqdn']}.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/#{node['fqdn']}.key"
nginx['ssl_ciphers'] = "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256"
nginx['ssl_prefer_server_ciphers'] = "on"
nginx['ssl_protocols'] = "TLSv1.2"
nginx['gzip_enabled'] = false
letsencrypt['enable'] = false

As you can see, I haven’t configured anything related to mattermost here, because, as I understand, this is related only to the omnibus gitlab mattermost that comes preinstalled with gitlab, as far as I understand.

2

Wondering if any of the docs here might be helpful? https://docs.gitlab.com/omnibus/gitlab-mattermost/

3

Hi,

Yes, I’ve already checked that out. The problem with that is that it explains how to set up gitlab mattermost, not mattermost independently. Even the section " Running GitLab Mattermost on its own server" still refers to installing mattermost through gitlab.

The weird thing is that it sometimes works and I’m not sure how. If I create a new app in gitlab and paste the information in mattermost, without checking and of the boxes, not even the ‘trust’ box, then for the first time I can see an ‘authorize’ confirmation in gitlab and I can log into mattermost through SSO. But that seems to work up to a point, when it stops working, and I don’t understand why.

4

Thanks @vinci - I realized I pasted the wrong doc link. Deploying GitLab SSO for Mattermost outside of the Omnibus package, we have documentation available here: https://docs.mattermost.com/deployment/sso-gitlab.html

There are a few steps to configure on the Mattermost side as well

5

Hi,

All that’s described in the documentation I’ve added in system console - gitlab in the mattermost application. That translate to the directives in the config.json. There’s nothing much to it, it’s pretty straightforward.
The weird thing is that sometimes it does work, but most of the times I get this error and it doesn’t really make much sense.

6

Thanks for the followup. There have been some issues with self-signed certificates with GitLab SSO, but I’ve touched base with our team if they have any insight on the errors you’ve shared above.

7

Hi,
I happened to read about that too, but that’s not relevant in my case, as I am using valid certificates. The only special quality of these certificates is that they’re wild card certificates.
And again, why would it sometimes work, but other times not?

I’ve also taken into consideration a network problem.
Is there any way I cat set the timeout for the SSO, that is to say, how how it should wait for a response from gitlab after the POST request to /oauth/token? I’m not sure if that’s of any relevance, but who knows?

8

Hi,
I am not sure if I have found a pattern to it, but it seems that when I press the “Gitlab” button on the mattermost interface to use SSO, then I sign in, then I get the “Token request failed” error, I can choose 'Go back to mattermost" and then when I press the “Gitlab” button once more, it seems to work.
So the authentication is obviously being done, but something gets interrupted and I can’t for the life of me figure out what’s going on.

9

That’s strange to hear that it works as long as you’re already logged into GitLab. I don’t think that flow should be any different in that case, at least from the Mattermost side. Is there maybe something different in the GitLab logs between those two cases?

The only thing I can think is that it’s perhaps an issue with how GitLab’s LDAP login works with Mattermost. I know there have been issues with that in the past. We don’t officially support that setup since it overlaps without our paid version, but I’m not aware of any reason that it shouldn’t work.

10

Hi,

First of all it has nothing to do with LDAP. It doesn’t matter if I use local users of gitlab or the ldap users, the behaviour is identical. Gitlab has no problems with authenticating either way (local/ldap), so the only problem is related to how the authentication token reaches Mattermost.
(By the way, I get the auth token from gitlab from the mattermost container using curl, something to the effect of: curl --data “@auth.txt” --request POST https://repos.atenekom.eu/oauth/token)

Secondly, I am not sure you’ve understood the behaviour. Maybe I haven’t described it properly myself:

It makes no difference if you are already logged into gitlab or not. If you are already logged into gitlab, then you visit mattermost, you have to press “sign in with gitlab”. Then you are going to get the error “Token request failed”. Then you press “Back to mattermost”, then again “sign in with gitlab” and then it works. Every time the same pattern. I couldn’t find anything relevant on gitlab, and I think nothing reaches gitlab in the first place, 'cause the error is “timeout”, there’s no explicit error coming from gitlab.

So do you know of any way of increasing this timeout for the POST? Maybe it’s extremely short, I don’t know.

11

I’m writing the gitlab logs here, maybe they help debugging the problem.
“my.username” is the user I’m logging in with
“repos.example.com” is gitlab and “chat.example.com” is mattermost:

==> /var/log/gitlab/unicorn/unicorn_stdout.log <==
I, [2019-08-03T21:32:39.472968 #9258] INFO – omniauth: (ldapmain) Callback phase initiated.

==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST “/users/auth/ldapmain/callback” for 192.168.50.1 at 2019-08-03 21:32:39 +0000

==> /var/log/gitlab/nginx/gitlab_access.log <==
192.168.50.1 - - [03/Aug/2019:21:32:46 +0000] “POST /users/auth/ldapmain/callback HTTP/2.0” 302 417 “https://repos.example.com/users/sign_in” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36”
192.168.50.1 - - [03/Aug/2019:21:32:46 +0000] “GET /oauth/authorize?response_type=code&client_id=8b565c63bcb4e371ec87b8ffc1376ad851c0b0f6108114aa2e4538a14aab73fa&redirect_uri=https%3A%2F%2Fchat.example.com%2Fsignup%2Fgitlab%2Fcomplete&state=eyJhY3Rpb24iOiJsb2dpbiIsInRva2VuIjoiaG9pc2Fwd2d0eXl5ZTZjNjQxYm1tczk2bW00YmsxNTFkaWI5cHllN3h4YW9pam11cmtzZ2tuZm9ob3FudTFoNyJ9 HTTP/2.0” 302 318 “https://repos.example.com/users/sign_in” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36”

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{“method”:“POST”,“path”:“/users/auth/ldapmain/callback”,“format”:“html”,“controller”:“Ldap::OmniauthCallbacksController”,“action”:“ldapmain”,“status”:302,“duration”:349.43,“view”:0.0,“db”:95.91,“location”:“https://repos.example.com/oauth/authorize",“time”:“2019-08-03T21:32:46.319Z”,“params”:[{“key”:“utf8”,“value”:“✓”},{“key”:“authenticity_token”,“value”:“[FILTERED]”},{“key”:“username”,“value”:“my.username”},{“key”:“password”,“value”:“[FILTERED]”}],“remote_ip”:“192.168.50.1”,“user_id”:2,“username”:“my.username”,“ua”:"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36”,“queue_duration”:25.37,“correlation_id”:“fHIKfcWfrpa”}
{“method”:“GET”,“path”:“/oauth/authorize”,“format”:“html”,“controller”:“Oauth::AuthorizationsController”,“action”:“new”,“status”:302,“duration”:114.35,“view”:0.0,“db”:17.27,“location”:“https://chat.example.com/signup/gitlab/complete",“time”:“2019-08-03T21:32:46.566Z”,“params”:[{“key”:“response_type”,“value”:“code”},{“key”:“client_id”,“value”:“8b565c63bcb4e371ec87b8ffc1376ad851c0b0f6108114aa2e4538a14aab73fa”},{“key”:“redirect_uri”,“value”:“https://chat.example.com/signup/gitlab/complete”},{“key”:“state”,“value”:“eyJhY3Rpb24iOiJsb2dpbiIsInRva2VuIjoiaG9pc2Fwd2d0eXl5ZTZjNjQxYm1tczk2bW00YmsxNTFkaWI5cHllN3h4YW9pam11cmtzZ2tuZm9ob3FudTFoNyJ9”}],“remote_ip”:null,“user_id”:null,“username”:null,“ua”:null,“queue_duration”:null,“correlation_id”:"lnZydfLgwQ9”}

==> /var/log/gitlab/gitlab-rails/production.log <==
Processing by Ldap::OmniauthCallbacksController#ldapmain as HTML
Parameters: {“utf8”=>“✓”, “authenticity_token”=>“[FILTERED]”, “username”=>“my.username”, “password”=>“[FILTERED]”}
Redirected to https://repos.example.com/oauth/authorize?response_type=code&client_id=8b565c63bcb4e371ec87b8ffc1376ad851c0b0f6108114aa2e4538a14aab73fa&redirect_uri=https%3A%2F%2Fchat.example.com%2Fsignup%2Fgitlab%2Fcomplete&state=eyJhY3Rpb24iOiJsb2dpbiIsInRva2VuIjoiaG9pc2Fwd2d0eXl5ZTZjNjQxYm1tczk2bW00YmsxNTFkaWI5cHllN3h4YW9pam11cmtzZ2tuZm9ob3FudTFoNyJ9
Completed 302 Found in 349ms (ActiveRecord: 95.9ms | Elasticsearch: 0.0ms)
Started GET “/oauth/authorize?response_type=code&client_id=8b565c63bcb4e371ec87b8ffc1376ad851c0b0f6108114aa2e4538a14aab73fa&redirect_uri=https%3A%2F%2Fchat.example.com%2Fsignup%2Fgitlab%2Fcomplete&state=eyJhY3Rpb24iOiJsb2dpbiIsInRva2VuIjoiaG9pc2Fwd2d0eXl5ZTZjNjQxYm1tczk2bW00YmsxNTFkaWI5cHllN3h4YW9pam11cmtzZ2tuZm9ob3FudTFoNyJ9” for 192.168.50.1 at 2019-08-03 21:32:46 +0000
Processing by Oauth::AuthorizationsController#new as HTML
Parameters: {“response_type”=>“code”, “client_id”=>“8b565c63bcb4e371ec87b8ffc1376ad851c0b0f6108114aa2e4538a14aab73fa”, “redirect_uri”=>“https://chat.example.com/signup/gitlab/complete”, “state”=>“eyJhY3Rpb24iOiJsb2dpbiIsInRva2VuIjoiaG9pc2Fwd2d0eXl5ZTZjNjQxYm1tczk2bW00YmsxNTFkaWI5cHllN3h4YW9pam11cmtzZ2tuZm9ob3FudTFoNyJ9”}
Redirected to https://chat.example.com/signup/gitlab/complete?code=015125b7eb5e768a21108833f17b7307422c39daca60d3f7b8c83f06671d23f0&state=eyJhY3Rpb24iOiJsb2dpbiIsInRva2VuIjoiaG9pc2Fwd2d0eXl5ZTZjNjQxYm1tczk2bW00YmsxNTFkaWI5cHllN3h4YW9pam11cmtzZ2tuZm9ob3FudTFoNyJ9
Completed 302 Found in 114ms (ActiveRecord: 17.3ms | Elasticsearch: 0.0ms)

==> /var/log/gitlab/gitlab-rails/audit_json.log <==
{“severity”:“INFO”,“time”:“2019-08-03T21:32:46.274Z”,“correlation_id”:“fHIKfcWfrpa”,“author_id”:2,“entity_id”:2,“entity_type”:“User”,“with”:“ldapmain”,“target_id”:2,“target_type”:“User”,“target_details”:“Iacob, Vincentiu”}

==> /var/log/gitlab/gitlab-workhorse/current <==
{“correlation_id”:“fHIKfcWfrpa”,“duration”:6.875066871,“host”:“repos.example.com”,“level”:“info”,“method”:“POST”,“msg”:“access”,“proto”:“HTTP/1.1”,“referer”:“https://repos.example.com/users/sign_in",“remoteAddr”:“127.0.0.1:0”,“remoteIp”:“127.0.0.1”,“status”:302,“system”:“http”,“time”:“2019-08-03T21:32:46Z”,“uri”:“/users/auth/ldapmain/callback”,“userAgent”:"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36”,“written”:417}
{“correlation_id”:“lnZydfLgwQ9”,“duration”:0.127732058,“host”:“repos.example.com”,“level”:“info”,“method”:“GET”,“msg”:“access”,“proto”:“HTTP/1.1”,“referer”:“https://repos.example.com/users/sign_in",“remoteAddr”:“127.0.0.1:0”,“remoteIp”:“127.0.0.1”,“status”:302,“system”:“http”,“time”:“2019-08-03T21:32:46Z”,“uri”:“/oauth/authorize?response_type=code\u0026client_id=8b565c63bcb4e371ec87b8ffc1376ad851c0b0f6108114aa2e4538a14aab73fa\u0026redirect_uri=https%3A%2F%2Fchat.example.com%2Fsignup%2Fgitlab%2Fcomplete\u0026state=eyJhY3Rpb24iOiJsb2dpbiIsInRva2VuIjoiaG9pc2Fwd2d0eXl5ZTZjNjQxYm1tczk2bW00YmsxNTFkaWI5cHllN3h4YW9pam11cmtzZ2tuZm9ob3FudTFoNyJ9”,“userAgent”:"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36”,“written”:318}

==> /var/log/gitlab/nginx/gitlab_access.log <==
127.0.0.1 - - [03/Aug/2019:21:32:58 +0000] “GET /help HTTP/1.1” 200 90320 “” “curl/7.59.0-DEV”

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{“method”:“GET”,“path”:“/help”,“format”:“/”,“controller”:“HelpController”,“action”:“index”,“status”:200,“duration”:314.52,“view”:304.21,“db”:0.75,“time”:“2019-08-03T21:32:58.284Z”,“params”:,“remote_ip”:“127.0.0.1”,“user_id”:null,“username”:null,“ua”:“curl/7.59.0-DEV”,“queue_duration”:2.1,“correlation_id”:“xJmZvv2UrM7”}

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET “/help” for 127.0.0.1 at 2019-08-03 21:32:57 +0000
Processing by HelpController#index as /
Completed 200 OK in 314ms (Views: 304.2ms | ActiveRecord: 0.7ms | Elasticsearch: 0.0ms)

==> /var/log/gitlab/gitlab-workhorse/current <==
{“correlation_id”:“xJmZvv2UrM7”,“duration”:0.322560562,“host”:“localhost”,“level”:“info”,“method”:“GET”,“msg”:“access”,“proto”:“HTTP/1.1”,“referer”:“”,“remoteAddr”:“127.0.0.1:0”,“remoteIp”:“127.0.0.1”,“status”:200,“system”:“http”,“time”:“2019-08-03T21:32:58Z”,“uri”:“/help”,“userAgent”:“curl/7.59.0-DEV”,“written”:90228}

On the mattermost container I’m only getting the same line:

{“level”:“error”,“ts”:1564867969.842193,“caller”:“api4/oauth.go:493”,“msg”:“AuthorizeOAuthUser: Token request failed, Post https://repos.example.com/oauth/token: dial tcp: i/o timeout”}

There is also this 401 error that I get all the time, but I don’t think it is in any way related to the authentication:

192.168.50.1 - - [03/Aug/2019:21:32:50 +0000] “POST /plugins/com.mattermost.nps/api/v1/connected HTTP/2.0” 401 0 “-” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36” “-”

13

Okay, that’s useful to know that it happens with both GitLab users and LDAP ones. Mattermost shouldn’t handle those two differently, but that still helps to rule that out.

The timeout for that POST is unfortunately hardcoded, but it’s 3 seconds to connect and 30 seconds to finish, so I’d think that would be more than enough for a simple request. That’s also the standard for requests made by the Mattermost server, so I don’t know why it would change between the first and second login attempts. When you curl the address, I assume it doesn’t take longer than 3 seconds for the request to be made, does it?

Those logs look like what I’d expect if the login was working, except for the fact that there’s no entry in gitlab_access.log for /oauth/token which lines up with the timeout you’re seeing on the Mattermost side.

A couple questions since you’re using HTTPS for both servers:

  1. Are you using a self-signed certificate for either of them?
  2. Can you try enabling the Enable Insecure Outgoing Connections setting in Mattermost? This has helped solve similar problems in the past, but it’s obviously not ideal since it’s kind of invalidating using HTTPS in the first place.

Also, that 401 error is annoying, but acceptable. I’ll file a ticket for that since I know where it comes from though.

14

Hello,

Thank you for your answer.
I’m using on both server the same wildcard public certificate from digicert, which is acknowledged by basically all browsers. I would think the fact that it’s a wildcard certificate shouldn’t be a problem, but I’m not sure how the applications have been developed, of course.

I’ve changed the “EnableInsecureOutgoingConnections” directive to true, but it doesn’t make any difference. Whenever I connect, the first time it doesn’t work. Then I press “to back to mattermost”, then “sign in with gitlab”, then it works, of course.
BUT if I wait a little bit more before pressing ‘go back to mattermost’/‘sign in with gitlab’, it doesn’t work. I get the same “Token request failed” error after pressing “sign in with gitlab”. But then again, if I do it fast, it works… I can’t imagine where that comes from. But even that is not 100% consistent. It did work once also after waiting for a longer time. It doesn’t make any sense whatsoever.

15

I don’t think a widcard cert should cause problems here, and since it’s from a proper CA, I don’t think that would be the issue here. I’ll ask around to see if anyone else has any ideas of what might be causing it.

16

The only difference from the official instructions is the reverse proxy I’m using for mattermost. It’s an nginx that is slightly different, although I’ve observed the instructions for mattermost (regarding websockets and all that) and, of course, mattermost works just fine apart from this problem.
I’ve also installed gitlab with mattermost with self-signed certificate (without ldap - although, as I’ve already said, in this case I don’t think the ldap auth causes any troubles, as the problem occurs in the connection between gitlab and mattermost) on my own computer and after enabling insecure outgoing connections it works right away.

I should also mention that when I get the token error, it takes a little bit time for the mattermost page to load up. Then when I try again, it works fast.
On my own computer (on two different VMs - I’ve manually changed the host entries to match the domains) it works fast.

So I wouldn’t rule out a problem with that 3 seconds timeout. I’m not saying that it’s not sensible, but I think in this case I might need a little bit more in certain circumstances - unfortunately I don’t really know a lot about the network the VMs are installed on, except for the fact that there are sometimes network delays, as I’m accessing it through a VPN.
By the way, the response to the curl command is quite fast, but when I ping the gitlab vm from the mattermost vm first time, I do get a bit of a delay unfortunately, but by the second attept it works fine (I’m pinging it intentionally using the domain. That doesn’t seem to be a real problem either, because I’ve added the hosts manually in each container, but to no avail).

[later edit:]
Actually I think you should be cautious even about the speed with which it reacts. Sometimes it reacts pretty fast and I get the token error, so… :smiley:

17

It definitely sounds like it’s actually hitting the timeout then. I thought it might’ve been something else because we’ve seen other issues report timeouts in the past, but since you see the delay with the first ping sometimes, it looks like a real timeout.

I could see two ways of trying to solve this then:

  1. Try increasing the timeout in Mattermost. This would require code changes, and I’m not sure which timeout would be responsible since there’s quite a few different timeouts listed in the networking code. If you’re interested in trying this out, I can point you to our developer setup guide which is pretty good. changing the timeouts should be fairly easy since they’re all configured in one place.
  2. Figure out why the first request sometimes takes longer. You mentioned you don’t know much about the network that the VMs are on though, so it sounds like you only have limited access to them which would make this a lot harder.
18

Unfortunately this would simply take too long, even if I tried to do that, how would I upgrade it in production? Each time I’d have to make the changes in the code and then compile it? Not really nice.

Yes, it does make it much harder, indeed. I only have access to the virtual machines themselves, not to the hypervisor.