Webhook URL issue from Gitlab to Mattermost (SSL)

Troubleshooting
1

Hi all,

I’ve just installed SSL certificate for my Mattermost hosted in Google Cloud Platform, now mattermost works fine with no issue, but not the Webhook.

I’ve enabled Webhook for Mattermost and generated a webhook URL from it, and have my developer team to copy it inside Gitlab webhook settings.

Unfortunately, the error message below displayed:

Hook execution failed: SSL_connect returned=1 errno=0 state=error: certificate verify failed

and no webhook messages being sent to Mattermost channel.

For your information, I’ve webhook URL generated as following:
https://chat.3lc.my:8065/hooks/oxe4z3zaxiyqip11z6zgerdqyy
I’m using Letsencrypt SSL for my Mattermost domain.

And I have the following logs from my Gitlab:

=======================================
Started GET “/3LCsystems/webServices/services/slack/test” for 192.168.2.145 at 2017-09-01 14:06:16 +0800
Processing by Projects::ServicesController#test as HTML
Parameters: {“namespace_id”=>“3LCsystems”, “project_id”=>“webServices”, “id”=>“slack”}
Completed 500 Internal Server Error in 260ms (ActiveRecord: 6.3ms)

OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=error: certificate verify failed):
app/models/project_services/slack_service.rb:79:in execute' app/models/service.rb:116:in test’
app/controllers/projects/services_controller.rb:34:in test' lib/gitlab/request_profiler/middleware.rb:15:in call’
lib/gitlab/middleware/go.rb:16:in `call’

======================================

Is there something wrong with my configuration? Or is it caused by the SSL certificate itself? I’m beginner in Mattermost and SSL.

Hope someone can guide me on this, now all the webhooks are not working.

Thank you.
Best regards,
Kong

2

Hi @kongyc,

Thank you for your feedback,

Here are some docs that may help with your issue:

If these don’t help, please let us know and we can troubleshoot further.

3

Hi @lindy65,

Good day. I’ve tried the method as mentioned but the problem is, the webhook works fine if the webhook is connecting to my old Mattermost server (http); but same error message pop out if connecting to my new mattermost server (https) hosted at Google Cloud.

4

Hi @kongyc, have you tried to post an incoming webhook via curl like curl -i -X POST -d 'payload={"text": "Hello, this is some text\nThis is more text. :tada:"}' https://chat.3lc.my:8065/hooks/oxe4z3zaxiyqip11z6zgerdqyy?
I did try and seemed to work fine. I received a response like this after posting:

HTTP/1.1 200 OK
Content-Type: text/plain
X-Request-Id: ffaeq8z7c78mxbnri8psyfbujh
X-Version-Id: 4.0.0.4.0.4.e54a18c0d073f2cdefa1ffdab34ef7db.false
Date: Tue, 05 Sep 2017 12:46:03 GMT
Content-Length: 2

I assumed in your channel, it posted similar to what we have in the documentation - https://docs.mattermost.com/developer/webhooks-incoming.html

5

Could you check your Gitlab - Webhook setting and play around with Enable SSL verification at https://docs.gitlab.com/ce/user/project/integrations/webhooks.html#ssl-verification? See how it performed when enable or disable against your new Mattermost server (https).

6

Hi @saturnino,

Yes I received the notification in my Mattermost channel after the “curl” command being executed. But I also tried the “Enable SSL verification” at the gitlab project settings, it still trigger error messages:

  1. If I “Enable SSL verification”, the following error message shows:
    Hook execution failed: SSL_connect returned=1 errno=0 state=error: certificate verify failed

  2. If I Disable SSL verification, the following error message shows:
    Hook executed successfully but returned HTTP 400 {“id”=>“web.incoming_webhook.text.app_error”, “message”=>“No text specified”, “detailed_error”=>"", “request_id”=>“xbtgcniotpba7joy7qn6b86w3e”, “status_code”=>400}

Seems like the webhook is working from server end but not working on Gitlab application level ?

7

Hi,

Is it possible to check which certificate path is being checked during the execution of webhook testing from Gitlab to Mattermost? I believe it could be some configuration needed to be done in /etc/gitlab/gitlab.rb ? As I could get any clue while checking the error log generated in “gitlab-ctl tail” command.

8

Hi @kongyc, could you try dropping the port number (8065)? Try to setup your Mattermost Google Cloud like https://chat.3lc.my, then test your endpoint using curl -i -X POST -d 'payload={"text": "Hello, this is some text\nThis is more text. :tada:"}' https://chat.3lc.my/hooks/oxe4z3zaxiyqip11z6zgerdqyy. If you’re able to work it out, have it setup to your Gitlab and see if it works.

9

Hi @saturnino,

Thanks for the suggestion. I’ve setup my Mattermost as https://chat.3lc.my and tested using
curl -i -X POST -d 'payload={"text": "Hello, this is some text\nThis is more text. :tada:"}' https://chat.3lc.my/hooks/oxe4z3zaxiyqip11z6zgerdqyy

yes I can received the notification in my Mattermost channel, but when I setup the same webhook URL in my Gitlab, same error message displayed.

I checked in my Mattermost platform log, it show this:

TLS handshake error from 121.121.83.100:38374: remote error: tls: unknown certificate authority

Is it something wrong with my certificate? I’m using Letsencrypt SSL, which is stored in my Mattermost Google Cloud server /etc/letsencrypt/live/chat.3lc.my/ and there are 4 pem files there:

cert.pem
chain.pem
fullchain.pem
privkey.pem

Any other suggestion to workaround on this issue?

Thanks a lot

10

Not sure how, but If possible, you may try to regenerate letsencrypt cert and set it up again against https://chat.3lc.my.
I’ve seen your post at https://forum.gitlab.com/t/gitlab-webhook-url-not-working-on-https-ssl/9814/7 and a suggestion there might solve your problem.

11

Hi @kongyc,

I’m the same boat as you are. Were you able to figure it out? I would really appreciate if you can share it. Thank you in advance.

12

I can tell you arguably the biggest issue quite plainly, I attempted to open this URL, and the domain does not exist, as you can see below. I made a GET request by just trying to open it in chrome, as well as a POST request using reqbin.com and both return no domain.

image
image1366×697 24.7 KB

image
image1348×694 48.8 KB

Can you verify that the URL you provided is correct to the one that you created in your Mattermost instance and that this subdomain is correctly configured as well?

In your DNS provider for the TLD 3lc.my you need to add an additional A record for the domain chat.3lc.my that points to the public IP address of your Google Cloud server, and then the subdomain will exist. I personally, from a security standpoint, highly suggest using a reverse proxy as well, instead of directly calling the port 8065 in the URL. This is safer in terms of your data, but also allows for a much cleaner URL, as the webhook would be, for example, https://chat.3lc.my/hooks/oxe4z3zaxiyqip11z6zgerdqyy instead of https://chat.3lc.my:8065/hooks/oxe4z3zaxiyqip11z6zgerdqyy.
Let me know if this helps! :slight_smile: