Repeated TLS handshake errors in mattermost.log

Troubleshooting
1

Summary

Repeated TLS handshake errors in mattermost.log

Mattermost is working on SSL certificates generated with LetsEncrypt, but the machine is hosted on Lightsail and has been freezing requiring a reboot. I looked into the mattermost.log and saw this line repeated many times.

Jul 6 11:33:05 my-servers-ip mattermost[1145]: {“level”:“error”,“ts”:1594006385.0375116,“caller”:“http/server.go:3053”,“msg”:“http: TLS handshake error from w.x.y.z:23009: EOF”,“source”:“httpserver”}

My setup:
mattermost-team-5.23.1-linux.amd64
psql (PostgreSQL) 10.12 (Ubuntu 10.12-0ubuntu0.18.04.1)
Lightsail AWS instance

2

Hello, @brisance

May I know if you have the Forward80To443 parameter in config.json set to true or false by providing the output of the following command? Please remove any sensitive information if there are any.

cat /opt/mattermost/config/config.json | grep -A18 "ServiceSettings"

If it is set to false, can you please edit the value to true and observe if the error continues to appear in the mattermost.log?

3

Hello, thank you for responding. Here is the output of the command.
“ServiceSettings”: {
“SiteURL”: “https://my-mattermost-server”,
“WebsocketURL”: “”,
“LicenseFileLocation”: “”,
“ListenAddress”: “:443”,
“ConnectionSecurity”: “TLS”,
“TLSCertFile”: “”,
“TLSKeyFile”: “”,
“TLSMinVer”: “1.2”,
“TLSStrictTransport”: false,
“TLSStrictTransportMaxAge”: 63072000,
“TLSOverwriteCiphers”: ,
“UseLetsEncrypt”: true,
“LetsEncryptCertificateCacheFile”: “./config/letsencrypt.cache”,
“Forward80To443”: true,
“TrustedProxyIPHeader”: [
“X-Forwarded-For”,
“X-Real-IP”
],

Here is the the output from tail -f /var/log/syslog

Jul  7 15:54:33 my-mm-server's-ip mattermost[1150]: {"level":"error","ts":1594108473.2879639,"caller":"http/server.go:3053","msg":"http: TLS handshake error from an-ip-address:32551: EOF","source":"httpserver"}
Jul  7 15:54:33 my-mm-server's-ip mattermost[1150]: {"level":"error","ts":1594108473.4994977,"caller":"http/server.go:3053","msg":"http: TLS handshake error from an-ip-address:39403: EOF","source":"httpserver"}
Jul  7 15:54:33 my-mm-server's-ip mattermost[1150]: {"level":"error","ts":1594108473.718667,"caller":"http/server.go:3053","msg":"http: TLS handshake error from an-ip-address:28475: EOF","source":"httpserver"}
Jul  7 15:54:33 my-mm-server's-ip mattermost[1150]: {"level":"error","ts":1594108473.9341893,"caller":"http/server.go:3053","msg":"http: TLS handshake error from an-ip-address:21849: EOF","source":"httpserver"}
Jul  7 15:54:34 my-mm-server's-ip mattermost[1150]: {"level":"error","ts":1594108474.152642,"caller":"http/server.go:3053","msg":"http: TLS handshake error from an-ip-address:30069: EOF","source":"httpserver"}
Jul  7 15:54:37 my-mm-server's-ip mattermost[1150]: {"level":"error","ts":1594108477.3381143,"caller":"http/h2_bundle.go:4211","msg":"http2: server connection error from an-ip-address:52468: connection error: PROTOCOL_ERROR","source":"httpserver"}
Jul  7 15:54:55 my-mm-server's-ip mattermost[1150]: {"level":"error","ts":1594108495.898477,"caller":"http/h2_bundle.go:4211","msg":"http2: server connection error from an-ip-address:61393: connection error: PROTOCOL_ERROR","source":"httpserver"}
Jul  7 15:55:01 my-mm-server's-ip CRON[5573]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jul  7 15:55:51 my-mm-server's-ip mattermost[1150]: {"level":"error","ts":1594108551.3170328,"caller":"http/server.go:3053","msg":"http: TLS handshake error from an-ip-address:36841: EOF","source":"httpserver"}
Jul  7 15:55:51 my-mm-server's-ip mattermost[1150]: {"level":"error","ts":1594108551.491765,"caller":"http/server.go:3053","msg":"http: TLS handshake error from an-ip-address:61735: EOF","source":"httpserver"}
Jul  7 15:55:51 my-mm-server's-ip mattermost[1150]: {"level":"error","ts":1594108551.6650758,"caller":"http/server.go:3053","msg":"http: TLS handshake error from an-ip-address:63175: EOF","source":"httpserver"}
Jul  7 15:55:51 my-mm-server's-ip mattermost[1150]: {"level":"error","ts":1594108551.8388176,"caller":"http/server.go:3053","msg":"http: TLS handshake error from an-ip-address:44891: EOF","source":"httpserver"}
Jul  7 15:55:52 my-mm-server's-ip mattermost[1150]: {"level":"error","ts":1594108552.0122962,"caller":"http/server.go:3053","msg":"http: TLS handshake error from an-ip-address:56073: EOF","source":"httpserver"}
Jul  7 15:56:32 my-mm-server's-ip mattermost[1150]: {"level":"error","ts":1594108592.0275831,"caller":"http/server.go:3053","msg":"http: TLS handshake error from an-ip-address:27555: EOF","source":"httpserver"}
Jul  7 15:56:32 my-mm-server's-ip mattermost[1150]: {"level":"error","ts":1594108592.2052991,"caller":"http/server.go:3053","msg":"http: TLS handshake error from an-ip-address:22833: EOF","source":"httpserver"}
Jul  7 15:56:32 my-mm-server's-ip mattermost[1150]: {"level":"error","ts":1594108592.3827991,"caller":"http/server.go:3053","msg":"http: TLS handshake error from an-ip-address:54759: EOF","source":"httpserver"}
Jul  7 15:56:32 my-mm-server's-ip mattermost[1150]: {"level":"error","ts":1594108592.56038,"caller":"http/server.go:3053","msg":"http: TLS handshake error from an-ip-address:51253: EOF","source":"httpserver"}
Jul  7 15:56:32 my-mm-server's-ip mattermost[1150]: {"level":"error","ts":1594108592.7379246,"caller":"http/server.go:3053","msg":"http: TLS handshake error from an-ip-address:42675: EOF","source":"httpserver"}
J
4

Hi, @brisance

You are welcome. Since you mentioned that Mattermost is working with the SSL certificate, can I confirm if the instance is publicly available on the internet? If yes, can you run a SSL Server Test and confirm if there are any warnings or errors?

Additionally, has there been any loss of functionality after the Lightsail reboot?

While the error is generally related to the SSL certificate itself, can I please check if you have a proxy configured for the Mattermost instance? If yes, is it NGINX or a different one?

5

@ahmaddanial Here is the output from the Qualys test. https://cln.sh/akblR9

The site is accessible from the internet.

And here is the detailed report. https://cln.sh/aBKuGx

The MM server works normally after the Lightsail reboot.

I’m not sure how to check whether nginx is running as the proxy server.

When issuing the LetsEncrypt certs, I specified nginx as the webserver and was able to get the certificates installed, albeit with some minor issues. I forgot to turn off MM at that time but the SSL certs were installed successfully after turning it off temporarily.

6

Hi, @brisance

The report looks good. So, no issues with the SSL certificate.

Thanks for the confirmation on this. I am not entirely sure how to check for NGINX on Lightsail but I came across this old article - Quick start guide: Nginx on Amazon Lightsail. Maybe we can check with the system administrator responsible for deploying and installing Mattermost?

While the EOF error usually indicates that something is wrong with SSL / TLS configuration on Mattermost, I cannot tell just yet what is the underlying problem since you also mentioned that the instance is working normally after the lightsail reboot.

7

Maybe we can check with the system administrator responsible for deploying and installing Mattermost?

That would be me. :blush:

While the EOF error usually indicates that something is wrong with SSL / TLS configuration on Mattermost, I cannot tell just yet what is the underlying problem since you also mentioned that the instance is working normally after the lightsail reboot.

It is working correctly, except that it keeps writing these errors into syslog.