Problem with Gitlab/Mattermost on Centos 6 with SCL Apache 2.4

Troubleshooting
1

Recently enabled Mattermost on Gitlab (since updated to 8.5.5) which is configured to have both as virtual hosts on Apache. The installation is on CentOS 6.7 with SCL’d version of httpd24 which includes proxy_wstunnel_module.

Mattermost is working except for a problem with Web Sockets. I have found lots of comments and configurations about this but don’t seem to be able to get this working. Unsure if my configuration is incorrect or if this is a problem with using the SCL’d version of httpd24.

The configuration file for Apache is:

<VirtualHost *:80>
  LogLevel debug
  ServerName mattermost.example.com
  ServerAlias mattermost

  DocumentRoot /opt/gitlab/embedded/service/mattermost/web/

  ProxyPreserveHost On
  # ProxyRequests Off

  # Ensure that encoded slashes are not decoded but left in their encoded state.
  # http://doc.gitlab.com/ce/api/projects.html#get-single-project
  AllowEncodedSlashes NoDecode
  # AllowEncodedSlashes On

  RewriteEngine on

  RewriteCond %{REQUEST_URI}  ^/api/v1/websocket    [NC,OR]
  RewriteCond %{HTTP:UPGRADE} ^WebSocket$           [NC,OR]
  RewriteCond %{HTTP:CONNECTION} ^Upgrade$          [NC]
  RewriteRule .* ws://127.0.0.1:8065%{REQUEST_URI}  [P,QSA,L]

  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
  RewriteRule .* http://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]
  # RequestHeader set X-Forwarded-Proto "https"

  # Prevent apache from sending incorrect 304 status updates
  RequestHeader unset If-Modified-Since
  RequestHeader unset If-None-Match

  <Location /api/v1/websocket>
    Require all granted
    ProxyPassReverse ws://127.0.0.1:8065/api/v1/websocket
    ProxyPassReverseCookieDomain 127.0.0.1 mattermost.example.com
  </Location>

  <Location />
    Require all granted
    ProxyPassReverse http://127.0.0.1:8065/
    ProxyPassReverseCookieDomain 127.0.0.1 mattermost.example.com
  </Location>

  LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common_forwarded
  ErrorLog /var/log/httpd/mattermost.example.com_error.log
  CustomLog /var/log/httpd/mattermost.example.com_forwarded.log common_forwarded
  CustomLog /var/log/httpd/mattermost.example.com_access.log combined env=!dontlog
  CustomLog /var/log/httpd/mattermost.example.com.log combined

</VirtualHost>

But with this, I still get the warning message Please check connection, Mattermost unreachable. If issue persists, ask administrator to check WebSocket port. at the top of page when in Mattermost.

This is my first time with WebSockets and Apache, so am a little unsure as to what I should see in the logs, etc. In trying to resolve the problem, I tried accessing the api directly to check the responses and logs. If I use the URL http://mattermost.example.com/api/v1/websocket directly, i.e. not via Apache but through Gitlab, I get back:

Bad Request
{"id":"api.web_socket.connect.upgrade.app_error","message":"Failed to upgrade websocket connection","detailed_error":"","request_id":"ihsn1f8se3nmtfcx74ezq8omhr","status_code":500,"is_oauth":false}

But when going through Apache VirtualHost, I get back GitLab Mattermost needs your help: and the URL is adjusted to http://mattermost.example.com/ws:/127.0.0.1:8065/api/v1/websocket

In the logs I see:

"GET /ws://127.0.0.1:8065/api/v1/websocket?session_token_index=0 HTTP/1.1" 301

Any suggestions/guidance very much appreciated.