[SOLVED] Ubuntu 16.04 Nginx Mattermost TLS error

Hello,

I started installing on a Ubuntu 16.04 machine with NGINX a mattermost server.
So yesterday everything worked fine. I did the installation then I went through the configuration.
After a restart of the server mattermost stopped working.
Either I am getting simply an empty page or depending on which I restart first a Bad Gateway message.

In the logs I see messages like: http: TLS handshake error from <address>: tls: first record does not look like a TLS handshake

My config looks like this

{
"ServiceSettings": {
    "SiteURL": "https://<DOMAIN>",
    "LicenseFileLocation": "",
    "ListenAddress": ":8065",
    "ConnectionSecurity": "TLS",
    "TLSCertFile": "/etc/letsencrypt/live/<DOMAIN>/fullchain.pem",
    "TLSKeyFile": "/etc/letsencrypt/live/<DOMAIN>/privkey.pem",
    "UseLetsEncrypt": true,
    "LetsEncryptCertificateCacheFile": "./config/letsencrypt.cache",
    "Forward80To443": true,
    "ReadTimeout": 300,
    "WriteTimeout": 300,
    "MaximumLoginAttempts": 10,
    "GoroutineHealthThreshold": -1,
    "GoogleDeveloperKey": "",
    "EnableOAuthServiceProvider": false,
    "EnableIncomingWebhooks": true,
    "EnableOutgoingWebhooks": true,
    "EnableCommands": true,
    "EnableOnlyAdminIntegrations": true,
    "EnablePostUsernameOverride": false,
    "EnablePostIconOverride": false,
    "EnableAPIv3": true,
    "EnableLinkPreviews": false,
    "EnableTesting": false,
    "EnableDeveloper": false,
    "EnableSecurityFixAlert": true,
    "EnableInsecureOutgoingConnections": false,
    "EnableMultifactorAuthentication": false,
    "EnforceMultifactorAuthentication": false,
    "EnableUserAccessTokens": false,
    "AllowCorsFrom": "",
    "SessionLengthWebInDays": 30,
    "SessionLengthMobileInDays": 30,
    "SessionLengthSSOInDays": 30,
    "SessionCacheInMinutes": 10,
    "WebsocketSecurePort": 443,
    "WebsocketPort": 80,
    "WebserverMode": "gzip",
    "EnableCustomEmoji": false,
    "EnableEmojiPicker": true,
    "RestrictCustomEmojiCreation": "all",
    "RestrictPostDelete": "all",
    "AllowEditPost": "always",
    "PostEditTimeLimit": 300,
    "TimeBetweenUserTypingUpdatesMilliseconds": 5000,
    "EnablePostSearch": true,
    "EnableUserTypingMessages": true,
    "EnableChannelViewedMessages": true,
    "EnableUserStatuses": true,
    "ClusterLogTimeoutMilliseconds": 2000
},
"TeamSettings": {
    "SiteName": "Mattermost",
    "MaxUsersPerTeam": 50,
    "EnableTeamCreation": false,
    "EnableUserCreation": true,
    "EnableOpenServer": false,
    "RestrictCreationToDomains": "",
    "EnableCustomBrand": false,
    "CustomBrandText": "",
    "CustomDescriptionText": "",
    "RestrictDirectMessage": "any",
    "RestrictTeamInvite": "all",
    "RestrictPublicChannelManagement": "all",
    "RestrictPrivateChannelManagement": "all",
    "RestrictPublicChannelCreation": "all",
    "RestrictPrivateChannelCreation": "all",
    "RestrictPublicChannelDeletion": "all",
    "RestrictPrivateChannelDeletion": "all",
    "RestrictPrivateChannelManageMembers": "all",
    "UserStatusAwayTimeout": 300,
    "MaxChannelsPerTeam": 2000,
    "MaxNotificationsPerChannel": 1000,
    "TeammateNameDisplay": "username"
	    },
	    "SqlSettings": {
	        "DriverName": "postgres",
	        "DataSource": "<connectionURL>",
	        "DataSourceReplicas": [],
	        "DataSourceSearchReplicas": [],
	        "MaxIdleConns": 20,
	        "MaxOpenConns": 300,
	        "Trace": false,
	        "AtRestEncryptKey": "blah",
	        "QueryTimeout": 30
	    },
	    "LogSettings": {
	        "EnableConsole": true,
	        "ConsoleLevel": "INFO",
	        "EnableFile": true,
	        "FileLevel": "INFO",
	        "FileFormat": "",
	        "FileLocation": "",
	        "EnableWebhookDebugging": true,
	        "EnableDiagnostics": true
	    },
	    "PasswordSettings": {
	        "MinimumLength": 5,
	        "Lowercase": false,
	        "Number": false,
	        "Uppercase": false,
    "Symbol": false
	    },
	    "FileSettings": {
	        "EnableFileAttachments": true,
	        "EnableMobileUpload": true,
	        "EnableMobileDownload": true,
	        "MaxFileSize": 52428800,
	        "DriverName": "local",
	        "Directory": "./data/",
	        "EnablePublicLink": false,
	        "PublicLinkSalt": "blah",
	        "InitialFont": "luximbi.ttf",
	        "AmazonS3AccessKeyId": "",
	        "AmazonS3SecretAccessKey": "",
	        "AmazonS3Bucket": "",
	        "AmazonS3Region": "us-east-1",
	        "AmazonS3Endpoint": "s3.amazonaws.com",
	        "AmazonS3SSL": true,
	        "AmazonS3SignV2": false,
	        "AmazonS3SSE": false
	    },
	    "EmailSettings": {
	        "EnableSignUpWithEmail": true,
	        "EnableSignInWithEmail": true,
	        "EnableSignInWithUsername": true,
	        "SendEmailNotifications": true,
	        "RequireEmailVerification": true,
	        "FeedbackName": "",
	        "FeedbackEmail": "",
	        "FeedbackOrganization": "",
	        "EnableSMTPAuth": true,
	        "SMTPUsername": "<ADDRESS>",
	        "SMTPPassword": "<PW>",
	        "SMTPServer": "<MAILDOMAIN>",
	        "SMTPPort": "587",
	        "ConnectionSecurity": "STARTTLS",
	        "InviteSalt": "<SALT>",
	        "SendPushNotifications": false,
	        "PushNotificationServer": "",
	        "PushNotificationContents": "generic",
	        "EnableEmailBatching": false,
	        "EmailBatchingBufferSize": 256,
	        "EmailBatchingInterval": 30,
	        "SkipServerCertificateVerification": true,
	        "EmailNotificationContentsType": "full"
	    },
	    "RateLimitSettings": {
	        "Enable": false,
	        "PerSec": 10,
	        "MaxBurst": 100,
	        "MemoryStoreSize": 10000,
	        "VaryByRemoteAddr": true,
	        "VaryByHeader": ""
	    },
	    "PrivacySettings": {
	        "ShowEmailAddress": false,
	        "ShowFullName": true
	    },
	    "SupportSettings": {
	        "TermsOfServiceLink": "https://about.mattermost.com/default-terms/",
	        "PrivacyPolicyLink": "https://about.mattermost.com/default-privacy-policy/",
	        "AboutLink": "https://about.mattermost.com/default-about/",
	        "HelpLink": "https://about.mattermost.com/default-help/",
	        "ReportAProblemLink": "https://about.mattermost.com/default-report-a-problem/",
	        "AdministratorsGuideLink": "https://about.mattermost.com/administrators-guide/",
	        "TroubleshootingForumLink": "https://about.mattermost.com/troubleshooting-forum/",
	        "CommercialSupportLink": "https://about.mattermost.com/commercial-support/",
	        "SupportEmail": "feedback@mattermost.com"
},
"AnnouncementSettings": {
    "EnableBanner": false,
    "BannerText": "",
    "BannerColor": "#f2a93b",
    "BannerTextColor": "#333333",
    "AllowBannerDismissal": true
},
"GitLabSettings": {
    "Enable": false,
    "Secret": "",
    "Id": "",
    "Scope": "",
    "AuthEndpoint": "",
    "TokenEndpoint": "",
    "UserApiEndpoint": ""
},
"GoogleSettings": {
    "Enable": false,
    "Secret": "",
    "Id": "",
    "Scope": "profile email",
    "AuthEndpoint": "https://accounts.google.com/o/oauth2/v2/auth",
    "TokenEndpoint": "https://www.googleapis.com/oauth2/v4/token",
    "UserApiEndpoint": "https://www.googleapis.com/plus/v1/people/me"
},
"Office365Settings": {
    "Enable": false,
    "Secret": "",
    "Id": "",
    "Scope": "User.Read",
    "AuthEndpoint": "https://login.microsoftonline.com/common/oauth2/v2.0/authorize",
    "TokenEndpoint": "https://login.microsoftonline.com/common/oauth2/v2.0/token",
    "UserApiEndpoint": "https://graph.microsoft.com/v1.0/me"
},
"LdapSettings": {
    "Enable": false,
    "LdapServer": "",
    "LdapPort": 389,
    "ConnectionSecurity": "",
    "BaseDN": "",
    "BindUsername": "",
    "BindPassword": "",
    "UserFilter": "",
    "FirstNameAttribute": "",
    "LastNameAttribute": "",
    "EmailAttribute": "",
    "UsernameAttribute": "",
    "NicknameAttribute": "",
    "IdAttribute": "",
    "PositionAttribute": "",
    "SyncIntervalMinutes": 60,
    "SkipCertificateVerification": false,
    "QueryTimeout": 60,
    "MaxPageSize": 0,
    "LoginFieldName": ""
},
"ComplianceSettings": {
    "Enable": false,
    "Directory": "./data/",
    "EnableDaily": false
},
"LocalizationSettings": {
    "DefaultServerLocale": "de",
    "DefaultClientLocale": "de",
    "AvailableLocales": "de,en"
},
"SamlSettings": {
    "Enable": false,
    "Verify": true,
    "Encrypt": true,
    "IdpUrl": "",
    "IdpDescriptorUrl": "",
    "AssertionConsumerServiceURL": "",
    "IdpCertificateFile": "",
    "PublicCertificateFile": "",
    "PrivateKeyFile": "",
    "FirstNameAttribute": "",
    "LastNameAttribute": "",
    "EmailAttribute": "",
    "UsernameAttribute": "",
    "NicknameAttribute": "",
    "LocaleAttribute": "",
    "PositionAttribute": "",
    "LoginButtonText": "With SAML"
},
"NativeAppSettings": {
    "AppDownloadLink": "https://about.mattermost.com/downloads/",
    "AndroidAppDownloadLink": "https://about.mattermost.com/mattermost-android-app/",
    "IosAppDownloadLink": "https://about.mattermost.com/mattermost-ios-app/"
},
"ClusterSettings": {
    "Enable": false,
    "ClusterName": "",
    "OverrideHostname": "",
    "UseIpAddress": true,
    "UseExperimentalGossip": false,
    "ReadOnlyConfig": true,
    "GossipPort": 8074,
    "StreamingPort": 8075
},
"MetricsSettings": {
    "Enable": false,
    "BlockProfileRate": 0,
    "ListenAddress": ":8067"
},
"AnalyticsSettings": {
    "MaxUsersForStatistics": 2500
},
"WebrtcSettings": {
    "Enable": false,
    "GatewayWebsocketUrl": "",
    "GatewayAdminUrl": "",
    "GatewayAdminSecret": "",
    "StunURI": "",
    "TurnURI": "",
    "TurnUsername": "",
    "TurnSharedKey": ""
},
"ElasticsearchSettings": {
    "ConnectionUrl": "http://dockerhost:9200",
    "Username": "elastic",
    "Password": "changeme",
    "EnableIndexing": false,
    "EnableSearching": false,
    "Sniff": true,
    "PostIndexReplicas": 1,
    "PostIndexShards": 1
},
"DataRetentionSettings": {
    "Enable": false
},
"JobSettings": {
    "RunJobs": true,
    "RunScheduler": true
},
  "PluginSettings": {
    "Plugins": {}
}
}

My mattermost.config for Nginx in sites-available linked in sites-enabled:

upstream backend {
   server <IP>:8065;
}

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;

server {
   listen 443 ssl http2;
   listen [::]:443 ssl http2;
   server_name <URL>;
 
   include /etc/nginx/conf.d/ssl.conf;

   location ~ /api/v[0-9]+/(users/)?websocket$ {
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       client_max_body_size 50M;
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options "SAMEORIGIN";
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       proxy_read_timeout 600s;
       proxy_pass http://backend;
   }

   location / {
       client_max_body_size 50M;
       proxy_set_header Connection "";
       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Frame-Options "SAMEORIGIN";
       proxy_buffers 256 16k;
       proxy_buffer_size 16k;
       proxy_read_timeout 600s;
       proxy_cache mattermost_cache;
       proxy_cache_revalidate on;
       proxy_cache_min_uses 2;
       proxy_cache_use_stale timeout;
       proxy_cache_lock on;
       proxy_pass http://backend;
   }
}

I hope someone can help me.

Greetings.

Where is the rest of your nginx config? its missing the SSL part listed on https://docs.mattermost.com/install/install-ubuntu-1604.html#configuring-nginx-with-ssl-and-http-2

Ha yes. Forgot sorry. And noticed that I missed the include. But after adding it after server_name, it still has the same behaviour.

nginx.conf contains

    user www-data;
worker_processes auto;
pid /run/nginx.pid;

events {
worker_connections 768;
# multi_accept on;
}

http {

##
# Basic Settings
##

sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;

server_tokens off;
server_names_hash_bucket_size 64;
# server_name_in_redirect off;

include /etc/nginx/mime.types;
default_type application/octet-stream;

##
# SSL Settings
##
#ssl_protocols TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;

##
# Logging Settings
##

access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;

##
# Gzip Settings
##

gzip on;
gzip_disable "msie6";

# gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_buffers 16 8k;
# gzip_http_version 1.1;
gzip_types text/plain text/css text/js text/xml text/javascript application/json application/javascript application/x-javascript application/xml application/xml+rss application/rss-xml image/svg+xml;

##
# Virtual Host Configs
##

include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}


#mail {
#       # See sample authentication script at:
#       # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
#
#       # auth_http localhost/auth.php;
#       # pop3_capabilities "TOP" "USER";
#       # imap_capabilities "IMAP4rev1" "UIDPLUS";
#
#       server {
#               listen     localhost:110;
#               protocol   pop3;
#               proxy      on;
#       }
#
#       server {
#               listen     localhost:143;
#               protocol   imap;
#               proxy      on;
#       }
#}

and the included ssl.conf

ssl on;
ssl_certificate /etc/letsencrypt/live/<DOMAIN>/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/<DOMAIN>/privkey.pem;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:!aNULL:!eNULL:!EXPORT:!DSS:!DES:!RC4:!3DES:!MD5:!PSK';

This is getting confusing now, your first config doesn’t contain ssl on; neither the ones in your second reply. Can you edit one of those to contain the full set of changes you have done to your nginx, so we can ensure nothing is missing? Better yet, could you review the guide and your settings? Because as far as I can tell it’s lacking. The MM config looks fine, what version are u running?

So now I updated the mattermost.conf for Nginx so that it includes the include /etc/nginx/conf.d/ssl.conf directive.
And in the ssl.conf I put ssl on;
I pasted the whole nginx.conf

Still does not help.

Version: nginx version: nginx/1.10.3 (Ubuntu)

#sudo ./platform version

Version: 4.1.0
Build Number: 4.1.0
Build Date: Tue Aug 15 22:11:43 UTC 2017
Build Hash: 0033e3e37b12cb5d951d21492500d66a6abc472b
Build Enterprise Ready: true
DB Version: 4.1.0

EDIT: and like I said everything worked fine. Only the restart killed it somehow.
Could another server conf interfere?

Well if there was no changes in the config between the restart it shouldn’t affect it, no. But the missing ssl on; for example could, also did u restart or reload it after doing those changes?

Let’s go thru a few tests, are you able to access mattermost directly https://domain:8065 and does it work or you get the SSL error there as well (if you have firewall blocking the port, temporally allow your IP to access it for debugging purpose)?

Could you post the latest error messages from both nginx logs and mattermost(kindly hide any sensitive information such as ip or domain names)?


Sorry missed you last edit, well it depends on what the other server configs are, but if they conflict with each other I am sure the nginx syntax would fail and not even start.

On another note, if you are registered to https://pre-release.mattermost.com/ feel free to mention me there on the ~peer-to-peer help channel :wink:

When I do lynx :8065
I am getting unsuppoerted URL scheme.

On laptop I get with http: �
with HTTPS: SSL_ERROR_INTERNAL_ERROR_ALERT

And the logs are not very verbose :frowning:

EDIT:
E.g. for nextcloud I have a nginx config file containing within the server tag

 # WARNING: Only add the preload option once you read about
     # the consequences in https://hstspreload.org/. This option
     # will add the domain to a hardcoded list that is shipped
     # in all major browsers and getting removed from this list
     # could take several months.
     add_header X-Content-Type-Options nosniff;
     add_header X-Frame-Options "SAMEORIGIN";
     add_header X-XSS-Protection "1; mode=block";
     add_header X-Robots-Tag none;
     add_header X-Download-Options noopen;
     add_header X-Permitted-Cross-Domain-Policies none;

Can this have a sideeffect?

So using a browser (chrome, firefox) and accessing the mattermost URL directly(not thru nginx proxy) you get a SSL alert error?

Can you try changing the TLSCertFile to the actual certificate instead of the fullpem?

It will be a file that start with a single certificate between ---- BEGIN CERTIFICATE --- and ---- END CERTIFICATE ---- or something similar.

I love browsers…

Ähm, I replaced it with the only other possible file which is named cert.pem.

Chrome gives me ERR_SSL_PROTOCOL_ERROR
Firefox shows same error as before.

I replaced it in the mattermost config.json
and then in the ssl.conf file.

We are only testing the mattermost server for now, so this change only needs to be done on mattermost.

In fact if I am not mistaken on nginx you will want to feed it the whole chain so that it can relay it to the browser as needed, so no changes to nginx for now.

That is why we are directly accessing the port 8065, if we can’t access mattermost directly, because of the SSL its most likely a issue with the mattermost config or the certificates.

@prixone Thanks for your help.

For all who might have similar issues. The logs weren’t very forthcoming. I only got weird smybols in the Browser.

Then I installed it anew which worked. Then step by step enabled the same settings.
In the end as I then suspected it has something to do with my via certbot from letsencrypt created certificats.
I still do not know what but I will figure this one out hopefully somewhen soon.