Mattermost behind Apache Proxy - wss not working

I’m new to Mattermost but not to server admin/web admin. I’ve got Mattermost up and running and I’m really liking it so far over the jumbled, complicated mess that is Riot/Synapse. Anyway - I’ve got it setup behind an Apache proxy. I’ve tried a few iterations here but none of them seem to want the wss traffic to pass through meaning I get the red banner warning saying “Please check connection, Mattermost unreachable. If issue persists, ask administrator to check WebSocket port.”

The way it’s setup now is that I’ve got my overall apache reverse proxy that has the following config:

<IfModule mod_ssl.c>

<VirtualHost *:80>

        ServerName chat.domain.net

        ProxyPreserveHost On

        ProxyPass / http://10.20.5.30:80
        ProxyPassReverse / http://chat.domain.net

        RewriteEngine on
        RewriteCond %{SERVER_NAME} =chat.domain.net
        RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

</VirtualHost>

        <VirtualHost *:443>

                ServerName chat.domain.net

                ProxyPreserveHost On
                SSLEngine On
                SSLProxyEngine On

                ProxyPass / https://10.20.5.30:443/
                ProxyPassReverse / https://chat.domain.net:443/

                RewriteEngine On
                RewriteCond %{REQUEST_URI} /api/v[0-9]+/(users/)?websocket [NC,OR]
                RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
                RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
                RewriteRule .* ws://10.20.5.30%{REQUEST_URI} [P,QSA,L]

                Include /etc/letsencrypt/options-ssl-apache.conf


                SSLCertificateFile /etc/letsencrypt/live/domain/cert.pem
                SSLCertificateKeyFile /etc/letsencrypt/live/domain/privkey.pem
                SSLCertificateChainFile /etc/letsencrypt/live/domain/chain.pem

        </VirtualHost>

</IfModule>

Then on the mattermost server I’ve got nginx running just fine with the following config:

upstream backend {
        server 10.20.5.30:8065;
        keepalive 32;
}

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;

server {
        listen 80;
        server_name    chat.domain.net;
        return 301 https://$server_name$request_uri;
}

server {
        listen 443 ssl http2;
        server_name chat.domain.net;

        ssl on;
        ssl_certificate /etc/nginx/ssl/00/fullchain.pem;
        ssl_certificate_key /etc/nginx/ssl/00/privkey.pem;
        ssl_session_timeout 1d;
        ssl_protocols TLSv1.2;
        ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
        ssl_prefer_server_ciphers on;
        ssl_session_cache shared:SSL:50m;

        location ~ /api/v[0-9]+/(users/)?websocket$ {
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                client_max_body_size 50M;
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header X-Frame-Options SAMEORIGIN;
                proxy_buffers 256 16k;
                proxy_buffer_size 16k;
                client_body_timeout 60;
                send_timeout 300;
                lingering_timeout 5;
                proxy_connect_timeout 90;
                proxy_send_timeout 300;
                proxy_read_timeout 90s;
                proxy_pass http://backend;
        }

        location / {
                client_max_body_size 50M;
                proxy_set_header Connection "";
                proxy_set_header Host $http_host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header X-Frame-Options SAMEORIGIN;
                proxy_buffers 256 16k;
                proxy_buffer_size 16k;
                proxy_read_timeout 600s;
                proxy_cache mattermost_cache;
                proxy_cache_revalidate on;
                proxy_cache_min_uses 2;
                proxy_cache_use_stale timeout;
                proxy_cache_lock on;
                proxy_http_version 1.1;
                proxy_pass http://backend;
        }
}

I’m not a big fan of having a proxy feed to a proxy but it’s working for everything but wss. When I turn off nginx on the mattermost box then it stops working completely so I haven’t tried eliminating the nginx proxy and just using the front end proxy only.

Do I need to eliminate the nginx proxy and just use the apache up front or is there a way to get apache to pass the wss on to the mattermost box.

@richardsonje This troubleshooting doc might help: Troubleshooting Mattermost Issues — Mattermost documentation

Been crazy busy but finally able to look around at this. The documentation you provided is just the troubleshooting steps I’ve already performed. I know it’s a proxy issue … I’m trying to figure out how to resolve the proxy issue.

Not sure if there’s a way to make it happen, however, I have it working now. I had to remove the nginx proxy from the mattermost VM and then I kept only the apache proxy on my primary reverse proxy. Once I got that setup, it works great now.

Hello @richardsonje

Can you please describe the complete solution? I’m in the same situation and been struggling for a while.
I keep getting the Bad Request error in the nginx access logs.

GET /api/v4/websocket HTTP/1.1" 400 193 

The interesting thing is that it’s working from time to time and the red top error disappears for a few minutes then it’s back.

What do you mean by “removing the nginx proxy from the mattermost VM”?
If I stop nginx proxy on the mattermost server I get the
Service Unavailable error.

Thanks for your help,
Istvan.

Hello @imokweb, I currently use Apache as a reverse proxy for my instance, and this is what my configuration file is, which runs both reverse proxies effectively:

<IfModule mod_ssl.c>
<VirtualHost *:443>
  # If you're not using a subdomain you may need to set a ServerAlias to:
  # ServerAlias www.mydomain.com
  ServerName XXXXXXXXXXURLXXXXXXXXX
  ServerAdmin XXXXXXXXADMIN-EMAILXXXXXXXX
  ProxyPreserveHost On
  DocumentRoot /opt/mattermost
  # Set web sockets
  ProxyRequests Off
RemoteIPHeader CF-Connecting-IP
RewriteEngine On
RewriteCond %{REQUEST_URI} ^/api/v3/users/websocket [NC,OR]
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
RewriteRule .* ws://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule .* http://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]
RequestHeader set X-Forwarded-Proto "https"

  <Location /api/v4/users/websocket>
    Require all granted
    ProxyPassReverse ws://127.0.0.1:8065/api/v4/users/websocket
    ProxyPassReverseCookieDomain 127.0.0.1 XXXXXXXXXXURLXXXXXXXXX
  </Location>

  <Location />
    Require all granted
    ProxyPassReverse https://127.0.0.1:8065/
    ProxyPassReverseCookieDomain 127.0.0.1 XXXXXXXXXXURLXXXXXXXXX
  </Location>
#  <Location /opt/chat_logs/.git>
#   Require all denied
#
#  </Location>
#Custom Error Pages
        ErrorDocument 401 /srv/error/401
        ErrorDocument 403 /srv/error/403
        ErrorDocument 404 /srv//error/404
        ErrorDocument 500 /srv//error/500


RedirectMatch 404 /\.git

Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/XXXXXXXXXXURLXXXXXXXXX/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/XXXXXXXXXXURLXXXXXXXXX/privkey.pem
SSLEngine on
        SSLCertificateFile /root/XXXXXXXXXXURLXXXXXXXXX.pem
        SSLCertificateKeyFile /root/XXXXXXXXXXURLXXXXXXXXX.key
</VirtualHost>
</IfModule>

Do note that I have replaced my website’s URL and support email information with placeholders. I hope thta this helps!

1 Like

This was also helpful for me and it solved this problem in my configuration:
https://docs.mattermost.com/configure/config-proxy-apache2.html

1 Like