Upgrading Breaks Push Notifications

Troubleshooting
1

I’ve posted about this in the past but that post is no longer getting any responses.

This problem is driving me to insanity and I don’t get what’s going on.

Setup:
Mattermost Host: CentOS 7 running MM 5.16.0
MySQL: Separate host
Proxy: CentOS 7 running nginx as the proxy

Problem:
As soon as I upgrade it breaks the notifications. Usually it’ll still send the banner notifications and the notifications to the lock screen but the badge indicator only works maybe 10% of the time.

Troubleshooting:
I’ve done everything in the aforementioned link. In addition, i’ve built an entirely new proxy that was originally an Apache proxy and replaced it with an nginx proxy.

I built a whole new VM for mattermost running Ubuntu 20.04 (just today). As soon as I point the proxy at the new server, the badge icon indicator breaks. All users but 1 are iOS. Problem persists regardless of OS. Every once in a while it’ll work, but there is seemingly no consistency. The banner notifications are also hit or miss - they’ll work and then just not work.

What am I missing here? I want to love Mattermost, i really do. And, other than this stupid issue, I love this platform. But … i’ve been chasing this for i think 2 years now and nothing. The absolute moment I switch over, it breaks.

Any additional ideas?

2

Hi, @richardsonje

Would you mind sharing the full detail of your environment:

  • config.json
  • NGINX configuration file (is it still the same as this one?)
  • Type of push notification server

I would be interested to know how the proxy breaks the badge icon as I personally have not encountered such issue before.

Another thing that I would like you to try is to sign up for the community.mattermost.com server and test if the unread badge breaks over there too. Thanks.

3

Sure no problem.

Type of Push Notification server:
https://push-test.mattermost.com

config.json file

{
“ServiceSettings”: {
“SiteURL”: “https://chat.domain.net”,
“WebsocketURL”: “”,
“LicenseFileLocation”: “”,
“ListenAddress”: “:8065”,
“ConnectionSecurity”: “”,
“TLSCertFile”: “”,
“TLSKeyFile”: “”,
“TLSMinVer”: “1.2”,
“TLSStrictTransport”: false,
“TLSStrictTransportMaxAge”: 63072000,
“TLSOverwriteCiphers”: ,
“UseLetsEncrypt”: false,
“LetsEncryptCertificateCacheFile”: “./config/letsencrypt.cache”,
“Forward80To443”: false,
“TrustedProxyIPHeader”: [
“X-Forwarded-For”,
“X-Real-IP”
],
“ReadTimeout”: 300,
“WriteTimeout”: 300,
“MaximumLoginAttempts”: 10,
“GoroutineHealthThreshold”: -1,
“GoogleDeveloperKey”: “AIzaSyDZl7Mp_6S–kF1h8e3ErRUqwFoFX7RhCs”,
“EnableOAuthServiceProvider”: false,
“EnableIncomingWebhooks”: true,
“EnableOutgoingWebhooks”: true,
“EnableCommands”: true,
“EnableOnlyAdminIntegrations”: true,
“EnablePostUsernameOverride”: false,
“EnablePostIconOverride”: false,
“EnableLinkPreviews”: true,
“EnableTesting”: false,
“EnableDeveloper”: false,
“EnableSecurityFixAlert”: true,
“EnableInsecureOutgoingConnections”: true,
“AllowedUntrustedInternalConnections”: “”,
“EnableMultifactorAuthentication”: false,
“EnforceMultifactorAuthentication”: false,
“EnableUserAccessTokens”: false,
“AllowCorsFrom”: “”,
“CorsExposedHeaders”: “”,
“CorsAllowCredentials”: false,
“CorsDebug”: false,
“AllowCookiesForSubdomains”: false,
“SessionLengthWebInDays”: 180,
“SessionLengthMobileInDays”: 180,
“SessionLengthSSOInDays”: 30,
“SessionCacheInMinutes”: 10,
“SessionIdleTimeoutInMinutes”: 43200,
“WebsocketSecurePort”: 443,
“WebsocketPort”: 80,
“WebserverMode”: “gzip”,
“EnableCustomEmoji”: true,
“EnableEmojiPicker”: true,
“EnableGifPicker”: true,
“GfycatApiKey”: “2_KtH_W5”,
“GfycatApiSecret”: “3wLVZPiswc3DnaiaFoLkDvB4X0IV6CpMkj4tf2inJRsBY6-FnkT08zGmppWFgeof”,
“RestrictCustomEmojiCreation”: “all”,
“RestrictPostDelete”: “all”,
“AllowEditPost”: “always”,
“PostEditTimeLimit”: -1,
“TimeBetweenUserTypingUpdatesMilliseconds”: 5000,
“EnablePostSearch”: true,
“MinimumHashtagLength”: 3,
“EnableUserTypingMessages”: true,
“EnableChannelViewedMessages”: true,
“EnableUserStatuses”: true,
“ExperimentalEnableAuthenticationTransfer”: true,
“ClusterLogTimeoutMilliseconds”: 2000,
“CloseUnusedDirectMessages”: false,
“EnablePreviewFeatures”: true,
“EnableTutorial”: true,
“ExperimentalEnableDefaultChannelLeaveJoinMessages”: true,
“ExperimentalGroupUnreadChannels”: “disabled”,
“ExperimentalChannelOrganization”: false,
“ImageProxyType”: “”,
“ImageProxyURL”: “”,
“ImageProxyOptions”: “”,
“EnableAPITeamDeletion”: false,
“ExperimentalEnableHardenedMode”: false,
“DisableLegacyMFA”: true,
“ExperimentalStrictCSRFEnforcement”: false,
“EnableEmailInvitations”: true,
“ExperimentalLdapGroupSync”: false,
“DisableBotsWhenOwnerIsDeactivated”: true,
“EnableBotAccountCreation”: true
},
“TeamSettings”: {
“SiteName”: “Name”,
“MaxUsersPerTeam”: 50,
“EnableTeamCreation”: true,
“EnableUserCreation”: true,
“EnableOpenServer”: false,
“EnableUserDeactivation”: false,
“RestrictCreationToDomains”: “”,
“EnableCustomBrand”: true,
“CustomBrandText”: “”,
“CustomDescriptionText”: “”,
“RestrictDirectMessage”: “any”,
“RestrictTeamInvite”: “all”,
“RestrictPublicChannelManagement”: “all”,
“RestrictPrivateChannelManagement”: “all”,
“RestrictPublicChannelCreation”: “all”,
“RestrictPrivateChannelCreation”: “all”,
“RestrictPublicChannelDeletion”: “all”,
“RestrictPrivateChannelDeletion”: “all”,
“RestrictPrivateChannelManageMembers”: “all”,
“EnableXToLeaveChannelsFromLHS”: false,
“UserStatusAwayTimeout”: 300,
“MaxChannelsPerTeam”: 2000,
“MaxNotificationsPerChannel”: 1000000,
“EnableConfirmNotificationsToChannel”: true,
“TeammateNameDisplay”: “username”,
“ExperimentalViewArchivedChannels”: false,
“ExperimentalEnableAutomaticReplies”: false,
“ExperimentalHideTownSquareinLHS”: false,
“ExperimentalTownSquareIsReadOnly”: false,
“ExperimentalPrimaryTeam”: “”,
“ExperimentalDefaultChannels”:
},
“ClientRequirements”: {
“AndroidLatestVersion”: “”,
“AndroidMinVersion”: “”,
“DesktopLatestVersion”: “”,
“DesktopMinVersion”: “”,
“IosLatestVersion”: “”,
“IosMinVersion”: “”
},
“SqlSettings”: {
“DriverName”: “mysql”,
“DataSource”: “mmadmin:supersecret@tcp(mysql:3306)/mattermost?charset=utf8mb4,utf8\u0026readTimeout=30s\u0026writeTimeout=30s”,
“DataSourceReplicas”: ,
“DataSourceSearchReplicas”: ,
“MaxIdleConns”: 20,
“ConnMaxLifetimeMilliseconds”: 3600000,
“MaxOpenConns”: 300,
“Trace”: false,
“AtRestEncryptKey”: “secret”,
“QueryTimeout”: 30
},
“LogSettings”: {
“EnableConsole”: true,
“ConsoleLevel”: “INFO”,
“ConsoleJson”: true,
“EnableFile”: true,
“FileLevel”: “INFO”,
“FileJson”: true,
“FileLocation”: “”,
“EnableWebhookDebugging”: true,
“EnableDiagnostics”: true
},
“NotificationLogSettings”: {
“EnableConsole”: true,
“ConsoleLevel”: “INFO”,
“ConsoleJson”: true,
“EnableFile”: true,
“FileLevel”: “INFO”,
“FileJson”: true,
“FileLocation”: “”
},
“PasswordSettings”: {
“MinimumLength”: 10,
“Lowercase”: true,
“Number”: true,
“Uppercase”: true,
“Symbol”: true
},
“FileSettings”: {
“EnableFileAttachments”: true,
“EnableMobileUpload”: true,
“EnableMobileDownload”: true,
“MaxFileSize”: 52428800,
“DriverName”: “local”,
“Directory”: “./data/”,
“EnablePublicLink”: false,
“PublicLinkSalt”: “supersecret”,
“InitialFont”: “nunito-bold.ttf”,
“AmazonS3AccessKeyId”: “”,
“AmazonS3SecretAccessKey”: “”,
“AmazonS3Bucket”: “”,
“AmazonS3Region”: “”,
“AmazonS3Endpoint”: “s3.amazonaws.com”,
“AmazonS3SSL”: true,
“AmazonS3SignV2”: false,
“AmazonS3SSE”: false,
“AmazonS3Trace”: false
},
“EmailSettings”: {
“EnableSignUpWithEmail”: true,
“EnableSignInWithEmail”: true,
“EnableSignInWithUsername”: true,
“SendEmailNotifications”: true,
“UseChannelInEmailNotifications”: false,
“RequireEmailVerification”: false,
“FeedbackName”: “Domain Alerts”,
“FeedbackEmail”: “alerts@domain.net”,
“ReplyToAddress”: “admin@domain.net”,
“FeedbackOrganization”: “”,
“EnableSMTPAuth”: true,
“SMTPUsername”: “postmaster@domain.net”,
“SMTPPassword”: “super super secret”,
“SMTPServer”: “smtp.mailgun.org”,
“SMTPPort”: “587”,
“ConnectionSecurity”: “STARTTLS”,
“SendPushNotifications”: true,
“PushNotificationServer”: “https://push-test.mattermost.com”,
“PushNotificationContents”: “generic”,
“EnableEmailBatching”: false,
“EmailBatchingBufferSize”: 256,
“EmailBatchingInterval”: 30,
“EnablePreviewModeBanner”: false,
“SkipServerCertificateVerification”: false,
“EmailNotificationContentsType”: “full”,
“LoginButtonColor”: “#0000”,
“LoginButtonBorderColor”: “#2389D7”,
“LoginButtonTextColor”: “#2389D7”
},
“RateLimitSettings”: {
“Enable”: false,
“PerSec”: 10,
“MaxBurst”: 100,
“MemoryStoreSize”: 10000,
“VaryByRemoteAddr”: true,
“VaryByUser”: false,
“VaryByHeader”: “”
},
“PrivacySettings”: {
“ShowEmailAddress”: true,
“ShowFullName”: true
},
“SupportSettings”: {
“TermsOfServiceLink”: “Terms of Use - Mattermost Open Source Collaboration Platform”,
“PrivacyPolicyLink”: “Mattermost Privacy Policy - Open Source Messaging”,
“AboutLink”: “Mattermost overview — Mattermost documentation”,
“HelpLink”: “Homepage | Mattermost Academy”,
“ReportAProblemLink”: “https://about.mattermost.com/default-report-a-problem/”,
“SupportEmail”: “feedback@mattermost.com”,
“CustomTermsOfServiceEnabled”: false,
“CustomTermsOfServiceReAcceptancePeriod”: 365
},
“AnnouncementSettings”: {
“EnableBanner”: false,
“BannerText”: “”,
“BannerColor”: “#f2a93b”,
“BannerTextColor”: “#333333”,
“AllowBannerDismissal”: true
},
“ThemeSettings”: {
“EnableThemeSelection”: true,
“DefaultTheme”: “default”,
“AllowCustomThemes”: true,
“AllowedThemes”:
},
“GitLabSettings”: {
“Enable”: false,
“Secret”: “”,
“Id”: “”,
“Scope”: “”,
“AuthEndpoint”: “”,
“TokenEndpoint”: “”,
“UserApiEndpoint”: “”
},
“GoogleSettings”: {
“Enable”: false,
“Secret”: “”,
“Id”: “”,
“Scope”: “profile email”,
“AuthEndpoint”: “Sign in - Google Accounts”,
“TokenEndpoint”: “https://www.googleapis.com/oauth2/v4/token”,
“UserApiEndpoint”: “https://people.googleapis.com/v1/people/me?personFields=names,emailAddresses,nicknames,metadata”
},
“Office365Settings”: {
“Enable”: false,
“Secret”: “”,
“Id”: “”,
“Scope”: “User.Read”,
“AuthEndpoint”: “Sign in to your account”,
“TokenEndpoint”: “Sign in to your account”,
“UserApiEndpoint”: “https://graph.microsoft.com/v1.0/me”
},
“LdapSettings”: {
“Enable”: false,
“EnableSync”: false,
“LdapServer”: “”,
“LdapPort”: 389,
“ConnectionSecurity”: “”,
“BaseDN”: “”,
“BindUsername”: “”,
“BindPassword”: “”,
“UserFilter”: “”,
“GroupFilter”: “”,
“GroupDisplayNameAttribute”: “”,
“GroupIdAttribute”: “”,
“FirstNameAttribute”: “”,
“LastNameAttribute”: “”,
“EmailAttribute”: “”,
“UsernameAttribute”: “”,
“NicknameAttribute”: “”,
“IdAttribute”: “”,
“PositionAttribute”: “”,
“LoginIdAttribute”: “”,
“SyncIntervalMinutes”: 60,
“SkipCertificateVerification”: false,
“QueryTimeout”: 60,
“MaxPageSize”: 0,
“LoginFieldName”: “”,
“LoginButtonColor”: “#0000”,
“LoginButtonBorderColor”: “#2389D7”,
“LoginButtonTextColor”: “#2389D7”
},
“ComplianceSettings”: {
“Enable”: false,
“Directory”: “./data/”,
“EnableDaily”: false
},
“LocalizationSettings”: {
“DefaultServerLocale”: “en”,
“DefaultClientLocale”: “en”,
“AvailableLocales”: “”
},
“SamlSettings”: {
“Enable”: false,
“EnableSyncWithLdap”: false,
“EnableSyncWithLdapIncludeAuth”: false,
“Verify”: true,
“Encrypt”: true,
“IdpUrl”: “”,
“IdpDescriptorUrl”: “”,
“AssertionConsumerServiceURL”: “”,
“ScopingIDPProviderId”: “”,
“ScopingIDPName”: “”,
“IdpCertificateFile”: “”,
“PublicCertificateFile”: “”,
“PrivateKeyFile”: “”,
“IdAttribute”: “”,
“FirstNameAttribute”: “”,
“LastNameAttribute”: “”,
“EmailAttribute”: “”,
“UsernameAttribute”: “”,
“NicknameAttribute”: “”,
“LocaleAttribute”: “”,
“PositionAttribute”: “”,
“LoginButtonText”: “SAML”,
“LoginButtonColor”: “#34a28b”,
“LoginButtonBorderColor”: “#2389D7”,
“LoginButtonTextColor”: “#ffffff”
},
“NativeAppSettings”: {
“AppDownloadLink”: “Host Mattermost on Your Own Infrastructure”,
“AndroidAppDownloadLink”: “https://about.mattermost.com/mattermost-android-app/”,
“IosAppDownloadLink”: “‎Mattermost on the App Store”
},
“ClusterSettings”: {
“Enable”: false,
“ClusterName”: “”,
“OverrideHostname”: “”,
“UseIpAddress”: true,
“UseExperimentalGossip”: false,
“ReadOnlyConfig”: true,
“GossipPort”: 8074,
“StreamingPort”: 8075,
“MaxIdleConns”: 100,
“MaxIdleConnsPerHost”: 128,
“IdleConnTimeoutMilliseconds”: 90000
},
“MetricsSettings”: {
“Enable”: false,
“BlockProfileRate”: 0,
“ListenAddress”: “:8067”
},
“ExperimentalSettings”: {
“ClientSideCertEnable”: false,
“ClientSideCertCheck”: “secondary”,
“EnableClickToReply”: false,
“LinkMetadataTimeoutMilliseconds”: 5000,
“RestrictSystemAdmin”: false
},
“AnalyticsSettings”: {
“MaxUsersForStatistics”: 2500
},
“ElasticsearchSettings”: {
“ConnectionUrl”: “http://localhost:9200”,
“Username”: “elastic”,
“Password”: “changeme”,
“EnableIndexing”: false,
“EnableSearching”: false,
“EnableAutocomplete”: false,
“Sniff”: true,
“PostIndexReplicas”: 1,
“PostIndexShards”: 1,
“ChannelIndexReplicas”: 1,
“ChannelIndexShards”: 1,
“UserIndexReplicas”: 1,
“UserIndexShards”: 1,
“AggregatePostsAfterDays”: 365,
“PostsAggregatorJobStartTime”: “03:00”,
“IndexPrefix”: “”,
“LiveIndexingBatchSize”: 1,
“BulkIndexingTimeWindowSeconds”: 3600,
“RequestTimeoutSeconds”: 30
},
“DataRetentionSettings”: {
“EnableMessageDeletion”: false,
“EnableFileDeletion”: false,
“MessageRetentionDays”: 365,
“FileRetentionDays”: 365,
“DeletionJobStartTime”: “02:00”
},
“MessageExportSettings”: {
“EnableExport”: false,
“ExportFormat”: “actiance”,
“DailyRunTime”: “01:00”,
“ExportFromTimestamp”: 0,
“BatchSize”: 10000,
“GlobalRelaySettings”: {
“CustomerType”: “A9”,
“SmtpUsername”: “”,
“SmtpPassword”: “”,
“EmailAddress”: “”
}
},
“JobSettings”: {
“RunJobs”: true,
“RunScheduler”: true
},
“PluginSettings”: {
“Enable”: true,
“EnableUploads”: true,
“EnableHealthCheck”: true,
“Directory”: “./plugins”,
“ClientDirectory”: “./client/plugins”,
“Plugins”: {
“antivirus”: {
“clamavhostport”: “localhost:3310”,
“scantimeoutseconds”: 10
},
“com.github.matterpoll.matterpoll”: {
“experimentalui”: false,
“trigger”: “poll”
},
“com.github.moussetc.mattermost.plugin.giphy”: {
“apikey”: “Z70J6MoPH9v6JvdYJWBlAe2oIzDFU7l9”,
“language (giphy only)”: “en”,
“provider”: “giphy”,
“rating”: “”,
“rendition”: “fixed_height”,
“renditiongfycat”: “gif100px”
},
“com.github.moussetc.mattermost.plugin.spoiler”: {
“integrationurl”: “”,
“spoilermode”: “redacted”
}
},
“PluginStates”: {
“antivirus”: {
“Enable”: false
},
“com.github.matterpoll.matterpoll”: {
“Enable”: true
},
“com.github.moussetc.mattermost.plugin.diceroller”: {
“Enable”: true
},
“com.github.moussetc.mattermost.plugin.giphy”: {
“Enable”: true
},
“com.github.moussetc.mattermost.plugin.spoiler”: {
“Enable”: true
},
“com.mattermost.custom-attributes”: {
“Enable”: true
},
“com.mattermost.draw-plugin”: {
“Enable”: true
},
“com.mattermost.nps”: {
“Enable”: true
},
“mattermost-file-list”: {
“Enable”: true
},
“memes”: {
“Enable”: false
}
}
},
“DisplaySettings”: {
“CustomUrlSchemes”: ,
“ExperimentalTimezone”: false
},
“ImageProxySettings”: {
“Enable”: false,
“ImageProxyType”: “local”,
“RemoteImageProxyURL”: “”,
“RemoteImageProxyOptions”: “”
}
}

Nginx Conf file

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;

upstream backend {
server 10.20.5.30:8065;
}

server {
if ($host = chat.domain.net.net) {
return 301 https://$host$request_uri;
} # managed by Certbot

    listen 80;
    server_name chat.domain.net.net;

    include snippets/letsencrypt.conf;
    return 301 https://$host$request_uri;

}

server {
listen 443 ssl http2;
server_name domain.net.net;
ssl_certificate /etc/letsencrypt/live/domain.net.net/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/domain.net.net/privkey.pem; # managed by Certbot
ssl_trusted_certificate /etc/letsencrypt/live/domain.net.net/chain.pem;
include snippets/ssl.conf;

    return 301 https://$host$request_uri;

}

server {
listen 443 ssl http2;
server_name chat.domain.net.net;
ssl_certificate /etc/letsencrypt/live/domain.net.net/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/domain.net.net/privkey.pem; # managed by Certbot
ssl_trusted_certificate /etc/letsencrypt/live/domain.net.net/chain.pem;
include snippets/ssl.conf;

    location ~ /api/v[0-9]+/(users/)?websocket$ {
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            client_max_body_size 50M;
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Frame-Options SAMEORIGIN;
            proxy_buffers 256 16k;
            proxy_buffer_size 16k;
            proxy_read_timeout 600s;
            proxy_pass http://backend;
    }

    location / {
            proxy_http_version 1.1;
            client_max_body_size 50M;
            proxy_set_header Connection "";
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Frame-Options SAMEORIGIN;
            proxy_buffers 256 16k;
            proxy_buffer_size 16k;
            proxy_read_timeout 600s;
            proxy_cache mattermost_cache;
  		 proxy_cache_revalidate on;
            proxy_cache_min_uses 2;
            proxy_cache_use_stale timeout;
            proxy_cache_lock on;
            proxy_pass http://backend;
    }

}

By “breaks the badge icon” i simply mean it not longer provides an unread count. Every once in a while, and it’s completely random, it will put a count of 1 even if there are 20 unread messages, but that’s all it’ll do. However, if I keep it at 5.16 it’ll count perfectly. I’ve got both the new Ubuntu 20.04 VM (5.27) and the existing CentOS 7 (5.16) running side by side. If I point the server to the Ubuntu box, no badge count. if I point it to the Centos one, works great. If you had a moment to read through the previous post, I’ve attempted this several different ways (upgrading the existing install) and none of it changes the end result.

I’ll get an account setup and do testing on the mattermost servers.

1 Like
4

Here’s an interesting update. I setup two community.mattermost.con accounts and logged into one on my phone and one on the desktop.

The first notification worked fine but after that they did the same thing. However, I think their might have been an outage because I was having issues just logging in.

However, the interesting part is I have logged out of the community server on my app on my phone and back in to my server. I’m getting the Lock Screen notifications from the community server and I’m not logged into it anywhere. Even just now as I’m typing this I got a notification from the Developers channel and, again, I’m not logged into the community server at all.

5

Hello, @richardsonje

Based on the following finding:

Can you please confirm which version of iOS app are you running on? I would like to see if i can reproduce the issue with the community.mattermost.com as well as my vanilla 5.27.0 Mattermost instance.

Additionally, are you still facing issues with receiving push notifications on the community.mattermost.com as of now since you mentioned that the notification was working fine before it started acting up again?

6

Looks like i’m on 1.35.1 (Build 326)

Yes - it’s still doing it. It’s weird how it works too. So maybe every 4th or 5th time i use my phone I’ll see a batch of 4 notifications all from the community.mattermost.com server. It will be 4 notifications pop up on the lock screen and the corresponding badge indicator on the app icon. Then if I open it it’ll be opened into my MM server, however, it tends to have weird connection problems at this point so I have to close the app and reopen it and then it’s back to normal for a while.

It almost has to be some sort of back-end routing/DNS problem, right? Something doesn’t update and it’s routing notifications incorrectly? I’m just shooting in the dark here because it’s such a weird problem.

7

Hi @richardsonje

Since I have not encountered such issue with community.mattermost.com on my end ever since I signed up for it about 1 year ago:

  • iPhone X
  • iOS 14.0
  • Mattermost iOS app 1.35.1

I want to further narrow down if this is an issue with the device. Can you log in to the web browser and log out from every session for community.mattermost.com before logging into your mobile device only?

Ping me there @ahmad.danial and we can perform some tests to see how the push notification works.

Also, can you please confirm that you are now running on the latest version of Mattermost on your end?

8

Ok - i’m logged into the community.mattermost.com server now via a browser and what’s interesting is the only active session it’s seeing is the one I’m logged into via the browser.

I’m logging out of my server now on the phone and logging into the community.mattermost.com one.

9

I’ve had some time to troubleshoot this some more and i’m still having issues.

I built a completely new, from scratch, Ubuntu 20.04 instance, left the nginx reverse proxy in place, created new MySQL database.

I just pointed the reverse proxy from the working older instance to the newer, completely fresh and no data on it instance … and the badge icon still will not update. Then banner notifications and lock screen notifications work seemingly fine (although I didn’t send a ton of messages but everyone i did send did work) BUT the badge indicator is still not working. The moment I revert it back to the old instance it works perfectly.

Before I tried the clean install I tried exporting the bulk data from the old one and importing to the new one. That whole process worked but the badge count on the app icon still refuses to update.

When I setup the whole new environment, I even removed the app from my phone and reinstalled to make sure there wasn’t something cached there and that didn’t have an impact.

Is there some way that the public DNS could be impacting this? The only thing I can think of that I haven’t tried yet is changing my external IP address but I just don’t see how that solves this problem.

10

Hello, @richardsonje

I escalated this to the mobile team to get more input on this edge case. So what determines the unread badge is when the server is about to send the notification, it checks the database for the amount of mentions the user has and sets the badge property in the notification payload.

Server sends the notification to the proxy which in turns parses the notification payload to match the APNS format for iOS and the FCM format for Android. Then, it relays the notification to the platform notification service. The notification is then pushed by these services to the device (during this period, Mattermost has no control over the notifications).

Once the notification is delivered:

  • iOS badge is set in the icon automatically by the OS following the number set in the badge property.
  • Android has a notification receiver and the app sets the badge number following Android’s documentation using the same property from the payload. It is very important to note that not all Android launchers show the number, some of them just show a dot instead and some launcher show nothing but the notification in the system tray.

One way to quickly test this is to find a channel with mentions and mark the post with the mention as unread, this will send a notification to just set the badge number.

11

Ok that works.

So, i think the only constant through all of this might be the MySQL server. Do you think that could be causing that or am I reading into that incorrectly?

12

Hello, @richardsonje

Happy to know that worked. Circling back on the information that you have shared through out this thread, MySQL could be one of the reason. However, I want to know how you built the database because you referred to the official documentation and it causes the push notification to break, it could be a different component that contributes to the issue.

There has not been any reports (at least the ones that I am aware of) so far. So, I would be really interested to understand the root cause behind this.

13

Sorry, nothing has fixed the issue yet. Was saying “that works” to you escalating this. I’m here to provide whatever information is necessary.

I’m going to test the MySQL theory here this afternoon.

14

Hi, @richardsonje

Got it. Sorry for the initial confusion.

Please continue to test the push notifications on community.mattermost.com so we can confirm if the issue boils down potentially to the network / device that you are on. Ensure that you are not connected to other sessions too.

No issues on my end so far on iPhone X (iOS 14.1) on the latest 1.37.0. The push notifications went through properly and the unread badge counter shows the right number accordingly based on the notification settings I have configured within the app.

15

I know I’m jumping in on this one a bit late, but I was reading this thread and saw something that stood out to me as a variable worth checking into, and thought it would be worth mentioning.

@ahmaddanial You stated in a previous post on this thread that the push notifications are parsed by the proxy:

I was just wondering if there was any possibility that the proxy parser formatting or parsing engine was changed in the update, and could need to be updated in the proxy configuration in the Nginx (or other web server) configuration files, and if an error in the parsing/conversion to the APNS/FCM format could be preventing the notifications/badge updates from ever reaching the instance?

16

Hi, @XxLilBoPeepsxX

Took a step back to see if I miss out on anything.

Since https://push-test.mattermost.com is being used as the TPNS, I can think of either the database or the proxy that is causing the issue. If Richard can confirm that the push notification works for community.mattermost.com, probably your theory makes sense.

Do you have more details on your suspicion on the proxy parser formatting / parsing engine?

17

Sorry for the delay, craziness.

Ok … I just rebuilt the bulk of my infrastructure. I added a 10Gb switch, rebuild my core storage, built two completely new VMs, one for MySQL, one for MM. Got a completely new instance setup, no MySQL server setup, it all work s… and, you guessed it - the badge notifications don’t work.

I’m going to say we’re probably on to something with the proxy. THat’s the only thing I didn’t change but it’s really the only spot that makes sense. Here’s my current conf file on my proxy:

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;

upstream backend {
server 10.20.5.21:8065;
keepalive 32;
}

server {

    listen 80;
    server_name chat.domain.net;

    return 301 https://$host$request_uri;

}

server {
    listen 443 ssl http2;
    server_name chat.domain.net;

    http2_push_preload on; # Enable HTTP/2 Server Push

    ssl on;
    ssl_certificate /etc/letsencrypt/live/domain.net/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/domain.net/privkey.pem; # managed by Certbot

    ssl_session_timeout 1d;

    # Enable TLS versions (TLSv1.3 is required upcoming HTTP/3 QUIC).
    ssl_protocols TLSv1.2 TLSv1.3;

    # Enable TLSv1.3's 0-RTT. Use $ssl_early_data when reverse proxying to
    # prevent replay attacks.
    #
    # @see: http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_early_data
    # ssl_early_data on;

    ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:50m;
    # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
    add_header Strict-Transport-Security max-age=15768000;
    # OCSP Stapling ---
    # fetch OCSP records from URL in ssl_certificate and cache them
    ssl_stapling on;
    ssl_stapling_verify on;

    # add_header X-Early-Data $tls1_3_early_data;

    location ~ /api/v[0-9]+/(users/)?websocket$ {

            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            client_max_body_size 50M;
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Frame-Options SAMEORIGIN;
            proxy_buffers 256 16k;
            proxy_buffer_size 16k;
            client_body_timeout 60;
            send_timeout 300;
            lingering_timeout 5;
            proxy_connect_timeout 90;
            proxy_send_timeout 300;
            proxy_read_timeout 90s;
            proxy_pass http://backend;
    }

    location / {

            client_max_body_size 50M;
            proxy_set_header Connection "";
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Frame-Options SAMEORIGIN;
            proxy_buffers 256 16k;
            proxy_buffer_size 16k;
            proxy_read_timeout 600s;
            proxy_cache mattermost_cache;
            proxy_cache_revalidate on;
            proxy_cache_min_uses 2;
            proxy_cache_use_stale timeout;
            proxy_cache_lock on;
            proxy_http_version 1.1;
            proxy_pass http://backend;

    }

}

1 Like
18

Hi, @richardsonje

No worries. I was on a long leave too.

I compared your configuration with the standard NGINX configuration with SSL included in the documentation. Apart from the order of items such as the following:

upstream backend {
   server 10.20.5.21:8065;
   keepalive 32;
}

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;

I noticed that the return 301 is different compared to the one in the article which is configured to be like the following. Can you make the necessary change here?

server {
    listen 80;
    server_name chat.domain.net;
    return 301 https://$server_name$request_uri;
}

The ssl_early_data on was commented out. Was this intended?

# ssl_early_data on;

Apart from that, the configuration looks similar. We are just looking at the configuration based on the suspicion that you have for now. So, let us tackle these first.

1 Like
19

I made the changes to the conf file and now i get no notifications at all. I don’t even get the ones i was getting before (everything but the badge indication). I’m still doing some testing but all i did was change the 301 and the order of the proxy_cache line. I have the ssl_early_data disabled because i can’t seem to get that module to work. When i enable it it just tells me it’s an unknown module.

20

Ok turns out i had a completely unrelated issue with VMware and multipathing that coincidentally caused the MySQL server to go down right as I did these changes.

I’ve got it back running now. I migrated in the old data but even before that i’m still experiencing inconsistent notifications. I get some, then i’ll miss a few but it’s still consistent that i don’t get the badge indicator.