Push Notifications not working using TPNS

Troubleshooting
1

Summary

Not able to get push notifications working using TPNS

Steps to reproduce

Server configured for TPNS using Google Play / iOS apps, users configured for push notifications. Send a user a direct message or mention, push notification never received. Check log and see push error i/o timeout

Expected behavior

User should receive push notification on mobile

Observed behavior

User never receives push notification. Log shows server attempted to send push notification, but error shows i/o timeout
{"level":"error","ts":1528326990.5882158,"caller":"app/notification.go:767","msg":"Device push reported as error for UserId=83h3ac3ump8btmx5tu3iew697y SessionId=ascgyw3hktf5mnhyrpq3dgtcsr message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp: i/o timeout","user_id":"83h3ac3ump8btmx5tu3iew697y"}

Other notes

I can curl http://push-test.mattermost.com from the server and receive back <html><body>Mattermost Push Proxy</body></html>, but when I ping push-test.mattermost.com it just times out

Edit

What OS and version is the Mattermost server installed?

Ubuntu 16.04

What is your Mattermost server version?

4.10.1

Are you experiencing the issues with the browser webapp, if so which one?

n/a

Are you experiencing the issues with the Mattermost Desktop App, if so what version and OS?

n/a

Are you experiencing the issues with the Mattermost Mobile App, if so what version and OS?

Android v1.8.0 (Build 101) as well as iOS

Can you send a snippet of the Mattermost server logs around the time of the incident?

{"level":"error","ts":1528377906.7466958,"caller":"app/notification.go:767","msg":"Device push reported as error for UserId=97cs51ui7pyx983drb68m7f8xw SessionId=f4iwoef5ajryxjxnho7frjreba message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp: i/o timeout","user_id":"97cs51ui7pyx983drb68m7f8xw"}
{"level":"error","ts":1528377988.0946994,"caller":"app/notification.go:767","msg":"Device push reported as error for UserId=97cs51ui7pyx983drb68m7f8xw SessionId=f4iwoef5ajryxjxnho7frjreba message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp: i/o timeout","user_id":"97cs51ui7pyx983drb68m7f8xw"}
{"level":"error","ts":1528378622.523168,"caller":"app/notification.go:767","msg":"Device push reported as error for UserId=66cetriqz7nidxdczawd8h4nse SessionId=un56qp7ftintprpwypgm1pxm6h message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp: i/o timeout","user_id":"66cetriqz7nidxdczawd8h4nse"}
{"level":"error","ts":1528378632.5719802,"caller":"app/notification.go:767","msg":"Device push reported as error for UserId=66cetriqz7nidxdczawd8h4nse SessionId=un56qp7ftintprpwypgm1pxm6h message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp: i/o timeout","user_id":"66cetriqz7nidxdczawd8h4nse"}
{"level":"error","ts":1528378641.2676554,"caller":"app/notification.go:767","msg":"Device push reported as error for UserId=nwnaeexaepy9dji7e5495xofbw SessionId=prndz4m55trbuyd73rgwjb87cr message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp: i/o timeout","user_id":"nwnaeexaepy9dji7e5495xofbw"}
{"level":"error","ts":1528378641.7068503,"caller":"app/notification.go:767","msg":"Device push reported as error for UserId=66cetriqz7nidxdczawd8h4nse SessionId=un56qp7ftintprpwypgm1pxm6h message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp: i/o timeout","user_id":"66cetriqz7nidxdczawd8h4nse"}

Are you running Mattermost in a container and/or using container orchestration?

No container

Link to debug log

https://drive.google.com/open?id=1vV9t5c4wL1hmTQL6jYLnCYjrBoR4o0RE

2

Hi @sstoveld!

To give you the best help we can, can you help take a look at the following:

  1. Please provide any relevant additional information from this doc, including server version, etc.: https://docs.mattermost.com/process/support.html.

  2. Review this section on push notifications, along with steps to troubleshoot: https://docs.mattermost.com/administration/config-settings.html#push-notification-contents.

  3. Review these docs regarding TPNS:

Let me know if this helps!

3

Hi Amy, thanks for the response. I’ve edited my post with the requested information. Unfortunately the links provided did not help. There does not seem to be any documentation that I can find that is relevant to the error I am getting using TPNS. I am using the open source Team Edition, so the HPNS documentation doesn’t seem to fit. It doesn’t seem like it’s a configuration issue, as it seems everything matches up to how it should in the documentation, but it’s just timing out for some reason.

Edit

Also forgot to mention. Push notifications previously worked when I first set up Mattermost. One day they just stopped working without any changes being made to the server. I just upgraded from Mattermost 4.3 to 4.10 last night in hopes that there may have been a fix for the push notifications

Edit 2

I just received one single push notification on my phone today. The first I have received in months. A co-worker also said the same, he received a single push notification today, first in months. Strange that one single notification made it through today. Any ideas?

4

Hi @sstoveld! Thank you for the additional information!

I will be testing this today - if I can’t seem to reproduce this, I will ask an engineer to take a look at the logs that you have attached earlier.

5

Hi @sstoveld - Giving you an update here that I have asked a colleague to help take a look at the logs that you sent over earlier.

6

Hi Amy, thanks for the update. Let me know what you find out. Thanks!

7

Hi @sstoveld! Just wanted to ask one more additional question:

  1. Can you share a screenshot of what you see when you go to Account Settings -> Security -> View and logout of active sessions? Does it have the apps listed?
8

Hi Amy, yes I can see the apps listed

19%20AM

9

Hi @sstoveld! Giving you an update that I have asked our team and other community members to take a look at this but this far nobody hasn’t been able to figure out what the problem might be. I am currently working on researching more information regarding issues around push notifications and TPNS + asking my colleagues on any other tips / questions to help troubleshoot this.

11

Hey Amy, thanks for the update.

I’ve been noticing an increase in received push notifications over the last few days. I’ve received about 4 or 5 notifications so far this week, whereas I’ve gone months without receiving a single one before. We’re still not getting about 95% of the push notifications, though

12

@sstoveld No worries, in the meanwhile, for the Android app can you help see if installing and reinstalling the app makes any difference? We also just released v1.9 if you want to upgrade and see if it helps? (iOS v1.9 will be available within the next few days as well).

13

Hey Amy, just uninstalled, cleared cache and data, and reinstalled. Still does not appear to be working. I had my co-worker send me a series of direct messages, and I received 2 notifications. Most of them failed

Snip from the log below

{"level":"error","ts":1529523078.9775481,"caller":"app/notification.go:767","msg":"Device push reported as error for UserId=5parki9q3tnn3jbdfhn86g56qo SessionId=b3f8b4681irqimku15ctnfmwky message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp: i/o timeout","user_id":"5parki9q3tnn3jbdfhn86g56qo"}
{"level":"error","ts":1529523078.9781137,"caller":"app/notification.go:767","msg":"Device push reported as error for UserId=5parki9q3tnn3jbdfhn86g56qo SessionId=dk61qy7k1iydi8r3pwg6mhh14r message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp: i/o timeout","user_id":"5parki9q3tnn3jbdfhn86g56qo"}
{"level":"error","ts":1529523088.3866422,"caller":"api4/context.go:222","msg":"/api/v4/files/undefined/thumbnail:Context code=400 rid=kmr1j1se17frmxzcreencu6ncr uid=5parki9q3tnn3jbdfhn86g56qo ip=72.45.241.242 Invalid or missing file_id parameter in request URL [details: ]","user_id":"5parki9q3tnn3jbdfhn86g56qo"}
{"level":"error","ts":1529523108.4947615,"caller":"app/notification.go:767","msg":"Device push reported as error for UserId=ir4j5qkf9fdrppetznpiwzwtga SessionId=85zcy6rr3pg6dx9wxhwjdzm9ch message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp: i/o timeout","user_id":"ir4j5qkf9fdrppetznpiwzwtga"}
{"level":"error","ts":1529523125.623908,"caller":"app/notification.go:767","msg":"Device push reported as error for UserId=5parki9q3tnn3jbdfhn86g56qo SessionId=dk61qy7k1iydi8r3pwg6mhh14r message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp: i/o timeout","user_id":"5parki9q3tnn3jbdfhn86g56qo"}
{"level":"error","ts":1529523125.6244533,"caller":"app/notification.go:767","msg":"Device push reported as error for UserId=5parki9q3tnn3jbdfhn86g56qo SessionId=b3f8b4681irqimku15ctnfmwky message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp: i/o timeout","user_id":"5parki9q3tnn3jbdfhn86g56qo"}
{"level":"error","ts":1529523126.8021681,"caller":"app/notification.go:767","msg":"Device push reported as error for UserId=5parki9q3tnn3jbdfhn86g56qo SessionId=b3f8b4681irqimku15ctnfmwky message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp: i/o timeout","user_id":"5parki9q3tnn3jbdfhn86g56qo"}
{"level":"error","ts":1529523126.8022878,"caller":"app/notification.go:767","msg":"Device push reported as error for UserId=5parki9q3tnn3jbdfhn86g56qo SessionId=dk61qy7k1iydi8r3pwg6mhh14r message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp: i/o timeout","user_id":"5parki9q3tnn3jbdfhn86g56qo"}
{"level":"info","ts":1529523127.7319462,"caller":"app/notification.go:775","msg":"Device was reported as removed for UserId=5parki9q3tnn3jbdfhn86g56qo SessionId=dk61qy7k1iydi8r3pwg6mhh14r removing push for this session","user_id":"5parki9q3tnn3jbdfhn86g56qo"}
{"level":"info","ts":1529523127.73252,"caller":"app/notification.go:775","msg":"Device was reported as removed for UserId=5parki9q3tnn3jbdfhn86g56qo SessionId=dk61qy7k1iydi8r3pwg6mhh14r removing push for this session","user_id":"5parki9q3tnn3jbdfhn86g56qo"}
{"level":"info","ts":1529523127.7350724,"caller":"app/notification.go:775","msg":"Device was reported as removed for UserId=5parki9q3tnn3jbdfhn86g56qo SessionId=dk61qy7k1iydi8r3pwg6mhh14r removing push for this session","user_id":"5parki9q3tnn3jbdfhn86g56qo"}
{"level":"error","ts":1529523131.0529444,"caller":"app/notification.go:767","msg":"Device push reported as error for UserId=me3f6fxo17gn8kaeu3t7du9f5w SessionId=7kh3pbku8prizpgkkyna516ysr message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp: i/o timeout","user_id":"me3f6fxo17gn8kaeu3t7du9f5w"}
{"level":"error","ts":1529523143.782067,"caller":"app/notification.go:767","msg":"Device push reported as error for UserId=5parki9q3tnn3jbdfhn86g56qo SessionId=b3f8b4681irqimku15ctnfmwky message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp: i/o timeout","user_id":"5parki9q3tnn3jbdfhn86g56qo"}
{"level":"error","ts":1529523160.0853095,"caller":"app/notification.go:767","msg":"Device push reported as error for UserId=5parki9q3tnn3jbdfhn86g56qo SessionId=b3f8b4681irqimku15ctnfmwky message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp: i/o timeout","user_id":"5parki9q3tnn3jbdfhn86g56qo"}
14

@sstoveld Feedback from engineers:

They are guessing it could be a network issue on your end. Where is the server physically located in the world? Also, can you check if you have a proxy sitting in between?

Let me know any questions!

15

Hi @amy.blais

Strange… The server is in our cloud hosted data centre in Ashburn, Virginia. There is no proxy in between. Our office is VPN’d to the servers in that DC via Cisco Meraki Site-to-Site VPN, though. I don’t see how that could affect it, though. Do the engineers think that may possibly have any issues?

16

Hi @sstoveld - The engineers cannot determine / have no idea what might be causing the problem, so they suggested that it could be a problem with the network or proxy.

Perhaps as a next step you could join our nightly build server and do some testing with DMs to see how push notifications are working in that server?

There are also multiple channels you can join on our server to ask questions / share more details on this issue, such as Native Mobile Apps, Bugs and Peer-to-peer Help!

17

Hi @sstoveld
I encountered the same problem, i was on Mattermost team edition 5.19.3 and then upgraded to 5.24 but still do not receive any notification!
I have no proxy settings and can curl https://push-test.mattermost.com successfully. We ran the Mattermost as a container in the Kubernetes platform.
We didnt have any problem 3 weeks ago!
Please help me if you have any solution.

Bests.
Amir

1 Like
18

Hi @sstoveld
I fixed it!
The problem was about the time of our servers!
it deviated about 10 minutes and after sync the time, all notifs sent!

19

Hey @amir-khalili , sorry just saw this now, forgot all about this topic. I haven’t had notifications working for quite some time, since i created this issue.

I’ll check this out, I believe the timezone is likely off. That seemed to fix the issue for you?

1 Like
20

Hi,
Yes, the syncing time, solved my issue.

Hope it can help you.

Amir

21

I seem to have a similar issue, but the time is correct both in the host (UnraidOS) and the Docker image.