Mattermost-push-proxy net/http error. cannot send iOS notifications

Troubleshooting
1

Hi devs,

Recently I’ve deployed mattermost services stack. Previously I had troubles with iOS device token and some other stuff. But all that is resolved now. The only problem that I am facing right now is that push-proxy service is not sending notifications successfully.

Services versions:
mattermost-server: 5.15
mattermost-webapp: 5.15
mattermost-mobile: 1.21
mattermost-push-proxy: 5.9.0

Below are the logs for push-proxy service:

Oct 04 13:25:45 ip-172-31-13-5 bash[1428]: 2019/10/04 13:25:45 INFO Loading /home/ubuntu/mattermost-push-proxy/config/mattermost-push-proxy.json
Oct 04 13:25:45 ip-172-31-13-5 bash[1428]: 2019/10/04 13:25:45 INFO Push proxy server is initializing...
Oct 04 13:25:45 ip-172-31-13-5 bash[1428]: 2019/10/04 13:25:45 INFO Initializing apple notification server for type=apple
Oct 04 13:25:45 ip-172-31-13-5 bash[1428]: 2019/10/04 13:25:45 INFO Initializing apple notification server for type=apple_rn
Oct 04 13:25:45 ip-172-31-13-5 bash[1428]: 2019/10/04 13:25:45 INFO Initializing Android notification server for type=android
Oct 04 13:25:45 ip-172-31-13-5 bash[1428]: 2019/10/04 13:25:45 INFO Initializing Android notification server for type=android_rn
Oct 04 13:25:45 ip-172-31-13-5 bash[1428]: 2019/10/04 13:25:45 INFO Server is listening on :8066
Oct 04 13:26:04 ip-172-31-13-5 bash[1428]: 2019/10/04 13:26:04 INFO Sending apple push notification for device=apple_rn and type=message
Oct 04 13:27:11 ip-172-31-13-5 bash[1428]: 2019/10/04 13:27:11 ERROR Failed to send apple push sid=8kij64tz47nidf64w5r1are4th did=071ac56dcaf43e52f8b72bd8afdfbe902a4b194d3755cf4dc77c6b7ac70d3e7d err=Post https://api.push.apple.com/3/device/071ac56dcaf43e52f8b72bd8afdfbe902a4b194d3755cf4dc77c6b7ac70d3e7d: net/http: request canceled (Client.Timeout exceeded while awaiting headers) type=apple_rn
Oct 04 13:40:51 ip-172-31-13-5 bash[1428]: 2019/10/04 13:40:51 INFO Sending apple push notification for device=apple_rn and type=message
Oct 04 13:41:59 ip-172-31-13-5 bash[1428]: 2019/10/04 13:41:59 ERROR Failed to send apple push sid=8kij64tz47nidf64w5r1are4th did=071ac56dcaf43e52f8b72bd8afdfbe902a4b194d3755cf4dc77c6b7ac70d3e7d err=Post https://api.push.apple.com/3/device/071ac56dcaf43e52f8b72bd8afdfbe902a4b194d3755cf4dc77c6b7ac70d3e7d: net/http: request canceled (Client.Timeout exceeded while awaiting headers) type=apple_rn
Oct 04 13:42:09 ip-172-31-13-5 bash[1428]: 2019/10/04 13:42:09 INFO Sending apple push notification for device=apple_rn and type=message
Oct 04 13:43:14 ip-172-31-13-5 bash[1428]: 2019/10/04 13:43:14 ERROR Failed to send apple push sid=8kij64tz47nidf64w5r1are4th did=071ac56dcaf43e52f8b72bd8afdfbe902a4b194d3755cf4dc77c6b7ac70d3e7d err=Post https://api.push.apple.com/3/device/071ac56dcaf43e52f8b72bd8afdfbe902a4b194d3755cf4dc77c6b7ac70d3e7d: net/http: request canceled (Client.Timeout exceeded while awaiting headers) type=apple_rn

I’ve hosted the server and the push-proxy server on same cloud machine. I’ve added the /push endpoint in nginx to forward the requests to push-proxy service. Android notifications are working perfectly.

This is my configuration file.

{
    "ListenAddress":":8066",
    "ThrottlePerSec":300,
    "ThrottleMemoryStoreSize":50000,
    "ThrottleVaryByHeader":"X-Forwarded-For",
    "EnableMetrics": false,
    "ApplePushSettings":[
        {
            "Type":"apple",
            "ApplePushUseDevelopment":false,
            "ApplePushCertPrivate":"./config/aps_production_priv.pem",
            "ApplePushCertPassword":"pass",
            "ApplePushTopic":"com.COMPANY.mattermost"
        },
        {
            "Type":"apple_rn",
            "ApplePushUseDevelopment":false,
            "ApplePushCertPrivate":"./config/aps_development_priv.pem",
            "ApplePushCertPassword":"pass",
            "ApplePushTopic":"com.COMPANY.mattermost"
        }
    ],
    "AndroidPushSettings":[
        {
            "Type":"android",
            "AndroidApiKey":"AAAA1iLDJ0c:APA91bFPaBBFvhHvf2DltR0yFnSLScxGYnDJyTlhciSeuHw9JdWF2aci7lI_LnesjwUDUK5LJXYACWMjaBeBDLEinCjr1QMVEvtd-vsFanGZyASRuePsekLebSktPd0pUKNibkD8OvGM"
        },
        {
            "Type":"android_rn",
            "AndroidApiKey":"AAAA1iLDJ0c:APA91bFPaBBFvhHvf2DltR0yFnSLScxGYnDJyTlhciSeuHw9JdWF2aci7lI_LnesjwUDUK5LJXYACWMjaBeBDLEinCjr1QMVEvtd-vsFanGZyASRuePsekLebSktPd0pUKNibkD8OvGM"
        }
    ]
}

Please help me.

2

@danialmalik One of our devs said that it seems that you are not able to reach Apple from your push proxy server, and suggested that you might want to start investigating from there.

3

Any updates on this ?

4

If I remember correctly, the reason behind my issue was that I was testing the notifications on an emulator. Once I deployed the app to flight and installed on a real device, things started working for me .

1 Like
5

Hello, @cosine99

If you are facing the same issue to @danialmalik, would you mind sharing the snippet of the mattermost.log and notifications.log when the push notification is expected to be sent out and delivered so we can dig into this further?

If you have not done it already, please enable the DEBUG file log level through the UI - System Console > Logging or through config.json before running the command below on the Mattermost Server terminal to capture the information that we need:

tail -f /opt/mattermost/logs/mattermost.log

Can you also confirm if you are able to reach out to the push notification server? For example:

Ahmad-Danials-MacBook-Pro:~ ahmaddanialmohammad$ echo | telnet push-test.mattermost.com 443
Trying 54.165.130.9...
Connected to push-test-mattermost-com-327239320.us-east-1.elb.amazonaws.com.
Escape character is '^]'.
Connection closed by foreign host.
6

Hello @ahmaddanial

Thanks for reaching out.
I am facing a different issue and hence, I created a new topic (Mattermost-push-proxy. Test notification using curl works, but DMs are not triggering notifs).

I’m able to send test notifications to my Android device, but DMs are not triggering notifications.
I’ve shared snippets of both the log files and included further information in the topic.
Kindly requesting your help on the same.