[Solved] Apache2 ReverseProxy with websocket <-> https

Troubleshooting
1

I’m currently trying to setup a mattermost-1.1.1 instance on an Ubuntu LTS 14, which was pretty straightforward.
I configured the service to listen to 127.0.0.1:8065 only and want to do a ReverseProxy via Apache2 on the same host using this configuration

< VirtualHost *:443>
SSLEngine on
SSLProtocol all -SSLv2
SSLHonorCipherOrder on
SSLCipherSuite “ECDHE-RSA-AES128-SHA256:AES128-GCM-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH”
Header add Strict-Transport-Security: “max-age=15768000;includeSubdomains”
SSLCompression Off
SSLCertificateChainFile /etc/apache2/ssl/zcore_ORG_chain.pem
SSLCertificateFile /etc/apache2/ssl/zcore.intra.pem
SSLCertificateKeyFile /etc/apache2/ssl/zcore.intra.key.pem
ServerName mattermost.zcore.intra
ServerSignature Off
ServerAlias mattermost
ProxyPreserveHost On
AllowEncodedSlashes NoDecode

< Location />
Require all granted
ProxyPass http:// mattermost.zcore.intra:8065
ProxyPassReverse http:// mattermost.zcore.intra:8065
</ Location>

LogFormat “%{X-Forwarded-For}i %l %u %t "%r" %>s %b” common_forwarded
ErrorLog /var/log/apache2/mattermost.zcore.intra/mattermost.zcore.intra_error.log
CustomLog /var/log/apache2/mattermost.zcore.intra/mattermost.zcore.intra_forwarded.log common_forwarded
CustomLog /var/log/apache2/mattermost.zcore.intra/mattermost.zcore.intra_access.log combined env=!dontlog
CustomLog /var/log/apache2/mattermost.zcore.intra/mattermost.zcore.intra.log combined

<VirtualHost *:80>
ServerName mattermost.zcore.intra
Redirect permanent / https:// mattermost.zcore.intra
</ VirtualHost>

The redirect works fine so far, but after a few seconds logged in I get the following message displayed on the site:

We cannot reach the Mattermost service. The service may be down or misconfigured. Please contact an administrator to make sure the WebSocket port is configured properly.

The corresponding server logs:

[2015/10/28 10:52:00 CET] [EROR] websocket connect err: websocket: could not find connection header with token ‘upgrade’
[2015/10/28 10:52:00 CET] [EROR] /api/v1/websocket:connect code=500 rid=emfn13fxjtnhpgp6nfrmoqdupw uid=f18pts15ib8xz8s4igctmm4ofa ip=192.168.3.128 Failed to upgrade websocket connection [details: ]

I guess there’s something missing in my configuration.

2

You need to allow the http connection to be upgraded to a websocket connection. Something like the following

{REQUEST_URI} ^/api/v1/websocket [NC,OR]
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
RewriteRule .* ws://127.0.0.1:8065%{REQUEST_URI}

see https://mattermost.atlassian.net/browse/PLT-635

3

After a whole night of digging I found a configuration that works-for-me™.

< VirtualHost *:443>
SSLEngine on
SSLProtocol all -SSLv2
SSLHonorCipherOrder on
SSLCipherSuite “ECDHE-RSA-AES128-SHA256:AES128-GCM-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH”
Header add Strict-Transport-Security: “max-age=15768000;includeSubdomains”
SSLCompression Off
SSLCertificateChainFile /etc/apache2/ssl/zcore_ORG_chain.pem
SSLCertificateFile /etc/apache2/ssl/zcore.intra.pem
SSLCertificateKeyFile /etc/apache2/ssl/zcore.intra.key.pem
ServerName mattermost.zcore.intra
ServerSignature Off
ServerAlias mattermost

ProxyPreserveHost On
ProxyRequests Off

RewriteEngine on
RewriteCond %{REQUEST_URI} ^/api/v1/websocket [NC,OR]
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
RewriteRule .* ws://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule .* http:// 127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]
RequestHeader set X-Forwarded-Proto “https”

< Location /api/v1/websocket>
Require all granted
ProxyPassReverse http:// 127.0.0.1:8065
ProxyPassReverseCookieDomain 127.0.0.1 mattermost.zcore.intra
< /Location>
< Location />
Require all granted
ProxyPassReverse http:// 127.0.0.1:8065
ProxyPassReverseCookieDomain 127.0.0.1 mattermost zcore.intra
< /Location>

LogFormat “%{X-Forwarded-For}i %l %u %t "%r" %>s %b” common_forwarded
ErrorLog /var/log/apache2/mattermost.zcore.intra/mattermost.zcore.intra_error.log
CustomLog /var/log/apache2/mattermost.zcore.intra/mattermost.zcore.intra_forwarded.log common_forwarded
CustomLog /var/log/apache2/mattermost.zcore.intra/mattermost.zcore.intra_access.log combined env=!dontlog
CustomLog /var/log/apache2/mattermost.zcore.intra/mattermost.zcore.intra.log combined
< /VirtualHost>
< VirtualHost *:80>
ServerName mattermost.zcore.intra
Redirect permanent / https:// mattermost.zcore.intra
< /VirtualHost>

Hope I didn’t miss anything.

Regs,
Rob

4

Unfortunately, this doesn’t seem to work, giving me the following error in logs:

[Wed Jun 15 17:00:39.470705 2016] [proxy:warn] [pid 16801] [client
x.x.x.x:57356] AH01144: No protocol handler was valid for the URL
/api/v3/users/websocket. If you are using a DSO version of mod_proxy,
make sure the proxy submodules are included in the configuration using
LoadModule.

I noted that it is now v3/users/websocket, so I modified my apache configuration as follows:

<VirtualHost x.x.x.x:443>
ServerName my.mattermost.org
SSLCertificateFile /etc/letsencrypt/live/my.mattermost.org/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/my.mattermost.org/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf

ProxyPreserveHost On
ProxyRequests Off

RewriteEngine On
RewriteCond %{REQUEST_URI} ^/api/v3/users/websocket [NC,OR]
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
RewriteRule .* ws://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule .* http://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]
RequestHeader set X-Forwarded-Proto "https"

  <Location /api/v3/users/websocket>
    Require all granted
    ProxyPassReverse ws://127.0.0.1:8065/api/v3/users/websocket
    ProxyPassReverseCookieDomain 127.0.0.1 my.mattermost.org
  </Location>

  <Location />
    Require all granted
    ProxyPassReverse https://127.0.0.1:8065/
    ProxyPassReverseCookieDomain 127.0.0.1 my.mattermost.org
  </Location>
</VirtualHost>

I initially experienced this problem as an inability to submit new messages, which I reported elsewhere, but now think simply presents a need to get a proper apache configuration instead.

5

OK, this has been resolved.

I resolved my issue using advice from this thread. In essence, It required the addition of the mod_proxy_wstunnel. In Jessie, I did it with:

sudo a2enmod proxy_wstunnel

The configuration is as above.