[SOLVED] Changing from HTTP to HTTPS results in ERR_CONNECTION_REFUSED for both Mattermost & GitLab

Changing the mattermost_external_url in my gitlab.rb to https instead of http (and then running sudo gitlab-ctl reconfigure) results in an ā€œERR_CONNECTION_REFUSEDā€ error in Chrome, for both Mattermost and GitLab. No idea why this would be happening, seems odd that itā€™s afffecting my GitLab instance as well.

Iā€™m using Ubuntu 15.04, and I have Letā€™s Encrypt set up for my GitLab instance. I havenā€™t generated any SSL certificates for my Mattermost instance yet.

Hereā€™s a screenshot of the error.

Any idea why this would be happening?

EDIT: I just checked my System Console, and I think this is the error itā€™s producing when itā€™s set to HTTPS: Failed to get security update information from Mattermost.

EDIT 2: I just had another thought about this: Could it be causing issues because Mattermost rules on a separate port, and trying to set it to HTTPS is trying to use the same port as GitLab?

Hi @revxx14,

Have you tried setting all the following in your gitlab.rb? Youā€™ll need another SSL certificate just for Mattermost.

mattermost_external_url 'https://mattermost.gitlab.example'
mattermost_nginx['redirect_http_to_https'] = true
mattermost_nginx['ssl_certificate'] = "/etc/gitlab/ssl/mattermost-nginx.crt"
mattermost_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/mattermost-nginx.key"
mattermost['service_use_ssl'] = true

This is documented here under the section ā€œRunning GitLab Mattermost with HTTPSā€. Let me know if that helps

Ah, no I did not. I think I see how to get this working, Iā€™ll report back if I make any progress.

I tried setting up SSL, but I was having issues because I donā€™t actually have DNS configured yet, Iā€™m just using a HOSTS file locally. I tried reverting back to HTTP, but now Iā€™m getting a 502 Bad Gateway error. It looks like Mattermost wonā€™t start.

Running gitlab-ctl status returns:

run: gitlab-workhorse: (pid 2309) 142s; run: log: (pid 737) 24887s
run: logrotate: (pid 2316) 142s; run: log: (pid 742) 24886s
down: mattermost: 0s, normally up, want up; run: log: (pid 734) 24887s
run: nginx: (pid 2336) 140s; run: log: (pid 736) 24887s
run: postgresql: (pid 2347) 139s; run: log: (pid 738) 24887s
run: redis: (pid 2349) 139s; run: log: (pid 735) 24887s
run: sidekiq: (pid 2364) 139s; run: log: (pid 739) 24887s
run: unicorn: (pid 2369) 138s; run: log: (pid 741) 24887s

As you can see, mattermost is down. If I try to run gitlab-ctl start mattermost, I get:

ok: run: mattermost: (pid 4263) 0s

But then running gitlab-ctl status mattermost, itā€™s still down:

down: mattermost: 1s, normally up, want up; run: log: (pid 734) 24954s

Iā€™m not sure what I changed that would be breaking this; I thought I reverted everything. The mattermost section of my gitlab.rb is here:

#####################
# GitLab Mattermost #
#####################

## Basic Settings
mattermost_external_url 'http://chat.mydomain.com'
mattermost['gitlab_enable'] = true
mattermost['gitlab_secret'] = "4**************************************************************0"
mattermost['gitlab_id'] = "c**************************************************************f"
mattermost['gitlab_scope'] = ""
mattermost['gitlab_auth_endpoint'] = "https://git.mydomain.com/oauth/authorize"
mattermost['gitlab_token_endpoint'] = "https://git.mydomain.com/oauth/token"
mattermost['gitlab_user_api_endpoint'] = "https://git.mydomain.com/api/v3/user"

## SMTP Settings
mattermost['email_feedback_name'] = "GitLab Mattermost",
mattermost['email_feedback_email'] = "git@mydomain.com",
mattermost['email_smtp_username'] = "git@mydomain.com",
mattermost['email_smtp_password'] = "************",
mattermost['email_smtp_server'] = "mydomain.com"
mattermost['email_smtp_port'] = "26"

## SSL Settings
# mattermost_nginx['redirect_http_to_https'] = true
# mattermost_nginx['ssl_certificate'] = "/etc/gitlab/ssl/mattermost-nginx.crt"
# mattermost_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/mattermost-nginx.key"
# mattermost['service_use_ssl'] = true

Could it be trying to use the same port as my HTTPS-enabled GitLab still?

no, mattermost doesnā€™t use any port, it just dies in initialization:[quote=ā€œrevxx14, post:4, topic:1123ā€]
down: mattermost: 0s, normally up, want up; run: log: (pid 734) 24887s
[/quote]
look into your mattermost logs and post the stack trace here.

Some of the init code is really wacky. I had the same problem after I put
mattermost['email_connection_security'] = "TLS"
in my gitlab.rb my mattermost got belly up in the same way as yours. I tracked it down to the email initialization panicing out. It works now with:
mattermost['email_smtp_port'] = 587 mattermost['email_connection_security'] = "STARTTLS"

If you donā€™t find the stack trace, just go back to your last stable config.rb and then change it only line by line so you can see which line is causing trouble.

1 Like

Thanks much! Itā€™s definitely something with the email. I commented out my email settings, ran gitlab-ctl reconfigure and chat is now working again. Iā€™ll double check the email settings and see if I can get that working again.

Okay, Iā€™ve definitely confirmed its something with the mail settings, but I canā€™t figure out what. Everything looks perfectly correct to me, and using the exact same settings via system console sends mail just fine. But when I try to configure it in gitlab.rb it crashes mattermost.

Hereā€™s what Iā€™m using in gitlab.rb, this all looks good, right? The names for each mattermost rule are correct?

## SMTP Settings
mattermost['email_feedback_name'] = "GitLab Mattermost",
mattermost['email_feedback_email'] = "git@mydomain.com", # removed for security
mattermost['email_smtp_username'] = "git@mydomain.com",  # removed for security
mattermost['email_smtp_password'] = "************",      # removed for security
mattermost['email_smtp_server'] = "mail.mydomain.com"    # removed for security
mattermost['email_smtp_port'] = "26"
mattermost['email_connection_security'] = ""

Hereā€™s what my cPanel says I need to use for email settings:

# Username:        git@mydomain.com
# Password:	   Use your cPanel password.
# Incoming Server: mail.mydomain.com
# IMAP Port:       143
# POP3 Port:       110
# Outgoing Server: mail.mydomain.com
# SMTP Port:       26
# Authentication is required for IMAP, POP3, and SMTP.

No idea what Iā€™m doing wrong. Password is correct, email is correct, port is correct, domains are correctā€¦ Very strange.

Not all combinations of these parameters are considered legal by the go smtp library :smiling_imp:
Youā€™ll need a stack trace to go on further. Search in the logs or try to execute the -version command with the CLI, which is incidentally the complete server, as described here

Iā€™ll bet my yesterdays socks that there is some null pointer panic and connectToSMTPServer or newSMTPClient is involved. There is definitely at least one null pointer check missing there. :worried:

Thanks. Iā€™m out of the office until Monday, but Iā€™ll try your suggestions then. I do remember seeing something about SMTP errors in the mattermost log.

EDIT: Actually I was able to see the logs via the system console, and this is the error Iā€™m getting:

[2016/03/23 02:30:53 CDT] [EROR] SMTP server settings do not appear to be configured properly err= details=dial tcp: lookup mail.mydomain.com on [::1]:53: read udp [::1]:33043->[::1]:53: read: connection refused

EDIT 2: Okay, so looking through the logs, Iā€™m seeing that I may have been using the wrong domain name for the email addressā€¦ Iā€™m an idiot :slight_smile: Hopefully fixing that on Monday will get it working again.

I finally got another chance to look at this, and I did get the email settings to work correctly. Iā€™m still having trouble setting up SSL, though, and Iā€™m not sure why. I literally make one change:

# Change
mattermost_external_url 'http://chat.mywebsite.com'
# To
mattermost_external_url 'https://chat.mywebsite.com'

And Nginx then refuses to start. No idea why this would be. Any suggestions?

Got it working. I think the key was that I had to generate the SSL key before changing the domain, and in order to do that, I had to add the following line:

mattermost_nginx['custom_gitlab_mattermost_server_config'] = "location ^~ /.well-known {\n alias /var/www/letsencrypt/.well-known;\n}\n"

Iā€™ve put together a rough tutorial to help anyone else whoā€™s stuck:

Setting up GitLab Mattermost with Letā€™s Encrypt

This guide assumes youā€™ve already installed GitLab along with GitLab Mattermost.

The first step is to switch to the root user as you need to run Letā€™s Encrypt as root:

sudi -i

Clone Letā€™s Encrypt

cd /root  
git clone https://github.com/letsencrypt/letsencrypt  

Create a mattermost.ini

Make sure youā€™re still in /root, and run the following:

mkdir letsencrypt-config
nano letsencrypt-config/mattermost.ini

Paste the following in to this configuration file:

# Let's Encrypt config for Mattermost

# Use the webroot authenticator
authenticator = webroot
webroot-path = /var/www/letsencrypt

# Use the standalone authenticator on port 443
# authenticator = standalone
# standalone-supported-challenges = tls-sni-01

# Generate certificates for the specified domain
domains = chat.yourdomain.com

# Register with the specified email address
email = youremail@yourdomain.com

# use a 4096 bit RSA key instead of 2048
rsa-key-size = 4096

Create the folder where authentication files will reside

mkdir -p /var/www/letsencrypt

Update gitlab.rb to alias chat.yourdomain.com/.well-known/ to /var/www/letsencrypt/

Edit your gitlab.rb:

nano /etc/gitlab/gitlab.rb

And add the following line:

mattermost_nginx['custom_gitlab_mattermost_server_config'] = "location ^~ /.well-known {\n alias /var/www/letsencrypt/.well-known;\n}\n"

Reconfigure GitLab to initialize the new setting:

gitlab-ctl reconfigure

Generate the Certificates

/root/letsencrypt/letsencrypt-auto certonly -c /root/letsencrypt-config/mattermost.ini

Update gitlab.rb to reflect live HTTPS settings

nano /etc/gitlab/gitlab.rb

Make the following changes:

mattermost_external_url 'https://chat.yourdomain.com'
mattermost['service_use_ssl'] = true
mattermost_nginx['redirect_http_to_https'] = true
mattermost_nginx['ssl_certificate'] = "/etc/letsencrypt/live/chat.yourdomain.com/fullchain.pem"
mattermost_nginx['ssl_certificate_key'] = "/etc/letsencrypt/live/chat.yourdomain.com/privkey.pem"

Then run:

gitlab-ctl reconfigure

Set up a cron job to auto update the certificates

nano /etc/cron.monthly/renew-ssl-certificates

Add the following:

#!/bin/bash

/root/.local/share/letsencrypt/bin/letsencrypt certonly -c /root/letsencrypt-config/mattermost.ini --renew-by-default

gitlab-ctl restart

And youā€™re done!

1 Like

I received the same msg ERR_CONNECTION_REFUSED and was blocked from doing any searches.
(But I could listen to and watch Youtube videos!)
I donā€™t know what your computer said, but I am running Google Chrome on windows 7 and above the message it said to:
1) Click on the Chrome Menu (the 2 bars in upper right hand corner of the Google Home page.
2) go to Settings > then "show advanced setting"
3) click Change proxy settings

  1. click LAN Settings
  2. UNclick "use a proxy server for your LAN"
    Was still getting ERR_CONNECTION_REFUSED
    Then closed all windows and programs
    REBOOTED PC
    Now, everything is fine! (REBOOTING (RESET) IS A NUISANCE, BUT IT HELPS)
    I hope this info helps you.
    http://www.deskdecode.com/err_connection_refused/

@revxx14 and @john, thanks for posting the tutorial and your solution! Iā€™ll mark this resolved for now.

I found a webpage from which i get rid out of this type of error issue, now you can also check this once for

getting rid of it completely from here: http://www.techinpost.com/fix-error-err_connection_refused-code-problem

Thanks for posting this information for others @Techinpost!

You can try with clear cookies method.

  • For example copy chrome://settings/clearbrowserData
  • Now hit enter, You will see pop up of clear browsing data.
  • After then just clear browser data and restart your browser.

You can try any form this link and also this link if need more ways.
Thanks