[Solved] Nginx Error Log: Connection refused

Troubleshooting
1

Hello everyone,

my Mattermost Instances are working well, but I found out that they produce a lot of erros in my nginx error logs. Most of thesse are messages state that the connection was refused and they refer to requests the uri /api/v4/users/status/ids.
To give you an impression:
2019/03/29 14:48:23 [error] 14213#14213: *3559 connect() failed (111: Connection refused) while connecting to upstream, client: some ip address , server: mysecondomain, request: "POST /api/v4/users/status/ids HTTP/2.0", upstream: "http://[::1]:8066/api/v4/users/status/ids", host: "mysecondomain"
2019/03/29 14:48:52 [error] 14213#14213: *1476 connect() failed (111: Connection refused) while connecting to upstream, client: some ip address , server: myfirstdomain, request: "POST /api/v4/users/status/ids HTTP/2.0", upstream: "http://[::1]:8065/api/v4/users/status/ids", host: "myfirstdomain"
2019/03/29 14:49:20 [error] 14213#14213: *3559 connect() failed (111: Connection refused) while connecting to upstream, client: some ip address , server: mysecondomain, request: "POST /api/v4/users/status/ids HTTP/2.0", upstream: "http://[::1]:8066/api/v4/users/status/ids", host: "mysecondomain"
2019/03/29 14:49:20 [error] 14213#14213: *3562 connect() failed (111: Connection refused) while connecting to upstream, client: some ip address , server: myfirstdomain, request: "POST /api/v4/users/status/ids HTTP/2.0", upstream: "http://[::1]:8065/api/v4/users/status/ids", host: "myfirstdomain"
2019/03/29 14:49:52 [error] 14213#14213: *1476 connect() failed (111: Connection refused) while connecting to upstream, client: some ip address , server: myfirstdomain, request: "POST /api/v4/users/status/ids HTTP/2.0", upstream: "http://[::1]:8065/api/v4/users/status/ids", host: "myfirstdomain"

I’m using Mattermost 5.9, Ubuntu 18.04.02 and my nginx version is 1.15.10.
I do not see anything peculiar in my Mattermost logs on the INFO level. They look fine.

Maybe it is related to my nginx configuration file. Here it is

upstream backend {
       server localhost:8065;
       keepalive 32;
    }

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;

server {
   listen 80 default_server;
   listen [::]:80 default_server;
   server_name chat.myfirstdomain.com;
   return 301 https://$server_name$request_uri;
}

server {
  listen 443 ssl http2;
  listen [::]:443 ssl http2;
  server_name  chat.myfirstdomain.com;

  ssl_certificate /path/to/certificate/chat.myfirstdomain.com/fullchain.pem;
  ssl_certificate_key /path/to/certificate/chat.myfirstdomain.com/privkey.pem;

  ssl_session_timeout 1d;
  ssl_protocols TLSv1.2 TLSv1.3;

  ssl_ciphers 'a list with my ciphers';
  ssl_dhparam /path/to/dhparams.pem;
  ssl_ecdh_curve secp521r1:secp384r1:prime256v1;


  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:50m;
  ssl_session_tickets off;
  ssl_trusted_certificate /path/to/certificate/chat.myfirstdomain.com/ca.pem;
  add_header Strict-Transport-Security "max-age=15768000; includeSubdomains;";

  ssl_stapling on;
  ssl_stapling_verify on;
  
  resolver ip address of systemmd-resolver; # needed for the OCSP stapling

   location ~ /api/v[0-9]+/(users/)?websocket$ {
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       client_max_body_size 50M;
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       client_body_timeout 60;
       send_timeout 300;
       lingering_timeout 5;
       lingering_timeout 5;
       proxy_connect_timeout 90;
       proxy_send_timeout 300;
       proxy_read_timeout 90s;
       proxy_pass http://backend;
   }

   location / {
       client_max_body_size 50M;
       proxy_set_header Connection "";
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options SAMEORIGIN;
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       proxy_read_timeout 600s;
       proxy_cache mattermost_cache;
       proxy_cache_revalidate on;
       proxy_cache_min_uses 2;
       proxy_cache_use_stale timeout;
       proxy_cache_lock on;
       proxy_http_version 1.1;
       proxy_pass http://backend;
   }
}

Does anyone have the same problems with his/her Mattermost Instance? I would like to stress, that my instances are working without any troubles. Does anyone have an idea why these error messages are created? And what I can do to avoid them?

Thank you very much!
Nils

2

Hi Nils,

If your Mattermost server is on a different machine from the Nginx proxy server you’ll need to change the Nginx config to point to that machine. For example:

upstream backend {
       server <Mattermost Server IP>:8065;
       keepalive 32;
}
3

Hi Paul,

thank you for your answer. The mattermost server is running on the same machine as the nginx webserver.
This did not cause the nginx error messages.
But I thought once more about the posted error messages, and figured that it has to do something with the following mattermost config entry:

"ListenAddress": "127.0.0.1:8065"

I removed 127.0.0.1 and the error messages stopped. Thus, my problem is solved.

But unfortunately, I am not very happy with changing this config entry, because it reopens the mattermost server for direct access to the public, i.e. it opened the port 8065 again. But I would like that all requests have to pass the nginx.

I am wondering if anyone knows how to configure nginx and mattermost in a way that all trafic has to pass the nginx server and that only the ports 80 and 443 need to be open?

Can it be that the regular expression in the nginx config file

location ~ /api/v[0-9]+/(users/)?websocket$

needs to be adapted? ( cf. with the above nginx error logs /api/v4/users/status/ids)

I don’t know. Does anyone have an idea?
Thank you very much,
Nils

4

Hi @nils-schween,

Using your configuration Mattermost only listens on IPv4 ( 127.0.0.1 ), but nginx’s upstream is localhost , which also resolves to an IPv6 address ( [::1] ). Did you try to use 127.0.0.1 as nginx upstream IP address?

5

Hi @amy.blais,

thank you very much Amy. This solved my problem! I did not know that localhost also resolves to the IPv6 [::1] address. With this knowledge at hand, the nginx error logs make way more sense to me, since they tell me that a request containing the IPv6 address is refused:

http://[::1]:8065/api/v4/users/status/ids

Once more thank you,
Nils

1 Like
6

@nils-schween, thank you for the solution. Where within nginx.conf do I replace nginx’s upstream “localhost” with “127.0.0.1”?

I can not find upstream in my nginx.conf

Thanks.

7

@amy.blais, thank you for the solution. Where within nginx.conf do I replace nginx’s upstream “localhost” with “127.0.0.1”?

I can not find upstream in my nginx.conf

Thanks.

8

HI @jkimathi,

if you take a look at the nginx configuration I posted, you find the upstream “backend” at the beginning,

upstream backend {
       server localhost:8065;
       keepalive 32;
    }

Replace localhost with 127.0.0.1.

Or alternatively, take a look at the nginx configuration, which can be found in the manual:
https://docs.mattermost.com/install/install-ubuntu-1804.html#configuring-nginx-as-a-proxy-for-mattermost-server

At its beginning you find the following lines:

upstream backend {
   server 10.10.10.2:8065;
   keepalive 32;
}

Replace 10.10.10.2 with 127.0.0.1.