[Solved] Problem with Gitlab/Mattermost on Centos 6 with SCL Apache 2.4

Recently enabled Mattermost on Gitlab (since updated to 8.5.5) which is configured to have both as virtual hosts on Apache. The installation is on CentOS 6.7 with SCL’d version of httpd24 which includes proxy_wstunnel_module.

Mattermost is working except for a problem with Web Sockets. I have found lots of comments and configurations about this but don’t seem to be able to get this working. Unsure if my configuration is incorrect or if this is a problem with using the SCL’d version of httpd24.

The configuration file for Apache is:

<VirtualHost *:80>
  LogLevel debug
  ServerName mattermost.example.com
  ServerAlias mattermost

  DocumentRoot /opt/gitlab/embedded/service/mattermost/web/

  ProxyPreserveHost On
  # ProxyRequests Off

  # Ensure that encoded slashes are not decoded but left in their encoded state.
  # http://doc.gitlab.com/ce/api/projects.html#get-single-project
  AllowEncodedSlashes NoDecode
  # AllowEncodedSlashes On

  RewriteEngine on

  RewriteCond %{REQUEST_URI}  ^/api/v1/websocket    [NC,OR]
  RewriteCond %{HTTP:UPGRADE} ^WebSocket$           [NC,OR]
  RewriteCond %{HTTP:CONNECTION} ^Upgrade$          [NC]
  RewriteRule .* ws://127.0.0.1:8065%{REQUEST_URI}  [P,QSA,L]

  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
  RewriteRule .* http://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]
  # RequestHeader set X-Forwarded-Proto "https"

  # Prevent apache from sending incorrect 304 status updates
  RequestHeader unset If-Modified-Since
  RequestHeader unset If-None-Match

  <Location /api/v1/websocket>
    Require all granted
    ProxyPassReverse ws://127.0.0.1:8065/api/v1/websocket
    ProxyPassReverseCookieDomain 127.0.0.1 mattermost.example.com
  </Location>

  <Location />
    Require all granted
    ProxyPassReverse http://127.0.0.1:8065/
    ProxyPassReverseCookieDomain 127.0.0.1 mattermost.example.com
  </Location>

  LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common_forwarded
  ErrorLog /var/log/httpd/mattermost.example.com_error.log
  CustomLog /var/log/httpd/mattermost.example.com_forwarded.log common_forwarded
  CustomLog /var/log/httpd/mattermost.example.com_access.log combined env=!dontlog
  CustomLog /var/log/httpd/mattermost.example.com.log combined

</VirtualHost>

But with this, I still get the warning message Please check connection, Mattermost unreachable. If issue persists, ask administrator to check WebSocket port. at the top of page when in Mattermost.

This is my first time with WebSockets and Apache, so am a little unsure as to what I should see in the logs, etc. In trying to resolve the problem, I tried accessing the api directly to check the responses and logs. If I use the URL http://mattermost.example.com/api/v1/websocket directly, i.e. not via Apache but through Gitlab, I get back:

Bad Request
{"id":"api.web_socket.connect.upgrade.app_error","message":"Failed to upgrade websocket connection","detailed_error":"","request_id":"ihsn1f8se3nmtfcx74ezq8omhr","status_code":500,"is_oauth":false}

But when going through Apache VirtualHost, I get back GitLab Mattermost needs your help: and the URL is adjusted to http://mattermost.example.com/ws:/127.0.0.1:8065/api/v1/websocket

In the logs I see:

"GET /ws://127.0.0.1:8065/api/v1/websocket?session_token_index=0 HTTP/1.1" 301

Any suggestions/guidance very much appreciated.

Bit more digging and I have found the following in the logs:

[Mon Mar 21 07:39:17.030403 2016] [authz_core:debug] [pid 28334] mod_authz_core.c(802): [client {ip_addr}:52922] AH01626: authorization result of Require all granted: granted
[Mon Mar 21 07:39:17.030458 2016] [authz_core:debug] [pid 28334] mod_authz_core.c(802): [client {ip_addr}:52922] AH01626: authorization result of <RequireAny>: granted
[Mon Mar 21 07:39:17.030522 2016] [proxy:debug] [pid 28334] mod_proxy.c(1100): [client {ip_addr}:52922] AH01143: Running scheme http handler (attempt 0)
[Mon Mar 21 07:39:17.030537 2016] [proxy_wstunnel:debug] [pid 28334] mod_proxy_wstunnel.c(328): [client {ip_addr}:52922] AH02450: declining URL http://mattermost.example.com/ws://127.0.0.1:8065/api/v1/websocket?session_token_index=0
[Mon Mar 21 07:39:17.030552 2016] [proxy_ajp:debug] [pid 28334] mod_proxy_ajp.c(708): [client {ip_addr}:52922] AH00894: declining URL http://mattermost.example.com/ws://127.0.0.1:8065/api/v1/websocket?session_token_index=0
[Mon Mar 21 07:39:17.030566 2016] [proxy_fcgi:debug] [pid 28334] mod_proxy_fcgi.c(944): [client {ip_addr}:52922] AH01076: url: http://mattermost.example.com/ws://127.0.0.1:8065/api/v1/websocket?session_token_index=0 proxyname: (null) proxyport: 0
[Mon Mar 21 07:39:17.030579 2016] [proxy_fcgi:debug] [pid 28334] mod_proxy_fcgi.c(950): [client {ip_addr}:52922] AH01077: declining URL http://mattermost.example.com/ws://127.0.0.1:8065/api/v1/websocket?session_token_index=0
[Mon Mar 21 07:39:17.030595 2016] [proxy:debug] [pid 28334] proxy_util.c(2020): AH00942: HTTP: has acquired connection for (*)
[Mon Mar 21 07:39:17.030611 2016] [proxy:debug] [pid 28334] proxy_util.c(2072): [client {ip_addr}:52922] AH00944: connecting http://mattermost.example.com/ws://127.0.0.1:8065/api/v1/websocket?session_token_index=0 to mattermost.example.com:80
[Mon Mar 21 07:39:17.031538 2016] [proxy:debug] [pid 28334] proxy_util.c(2194): [client {ip_addr}:52922] AH00947: connected /ws://127.0.0.1:8065/api/v1/websocket?session_token_index=0 to mattermost.example.com:80

Which has a declining URL http://mattermost.example.com/ws://127.0.0.1:8065/... which looks like it hasn’t switch protocols.

I’m having the same issue with Apache 2.4.6 and CentOS 7 with https.
“Please check connection, Mattermost unreachable. If issue persists, ask administrator to check WebSocket port.” message and all connections through wss:// return 301.

<VirtualHost *:443>
        ServerName mattermost.example.com
        ServerSignature Off

        # TLS
	SSLEngine on
        SSLProtocol all -SSLv2
        SSLHonorCipherOrder on
        Header add Strict-Transport-Security: "max-age=15768000;includeSubdomains"
        SSLCompression Off
        SSLCertificateKeyFile /.../privkey.pem
        SSLCertificateFile  /.../cert.pem
        SSLCertificateChainFile  /.../chain.pem

        ProxyPreserveHost On
        RewriteEngine     On
        SSLProxyEngine    On

        RewriteCond %{REQUEST_URI}  ^/api/v1/websocket      [NC,OR]
        RewriteCond %{HTTP:UPGRADE} ^WebSocket$             [NC,OR]
        RewriteCond %{HTTP:CONNECTION} ^Upgrade$            [NC]
        RewriteRule .* ws://127.0.0.1:8065%{REQUEST_URI}    [P,QSA,L]

        RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME}    !-f
        RewriteRule .* http://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]

        # Be sure to uncomment the next 2 lines if https is used
        RequestHeader set X-Forwarded-Proto "https"

        # Prevent apache from sending incorrect 304 status updates
        RequestHeader unset If-Modified-Since
        RequestHeader unset If-None-Match

        <Location /api/v1/websocket>
                Require all granted
                ProxyPassReverse ws://127.0.0.1:8065/api/v1/websocket
                ProxyPassReverseCookieDomain 127.0.0.1 mattermost.example.com
        </Location>

        <Location />
                Require all granted
                ProxyPassReverse http://127.0.0.1:8065/
                ProxyPassReverseCookieDomain 127.0.0.1 mattermost.example.com
        </Location>

        ProxyPreserveHost On
        ProxyRequests Off
		
</VirtualHost>

I tried a couple of further configuration changes, including compiling mod_proxy_wstunnel.c for Apache 2.2. Couldn’t get it working and could no longer spend any more time investigating. I have now moved the installation to CentOS 7 where it appears to be working.

Closed, but not resolved.

Since moving to CentOS 7, the only problem that I have noticed is “There appears to be a problem with your internet connection.” which appears occasionally. Still seems to work though.

Thanks. The issue was httpd. My CentOS 7 distribution was not totally up to date and there was a know issue with the old version of httpd and web socket proxy module.