The trouble with TPNS: dial tcp: i/o timeout

Troubleshooting
1

Summary
the trouble with TPNS: dial tcp: i/o timeout and it sometimes worked sometimes failed…

Steps to reproduce

  1. server env: GitHub - mattermost/mattermost-docker: Dockerfile for mattermost in production
    Mattermost Team Edition
    (Ver. E0)
    Mattermost Version: 5.31.6{PATCH}
    Database Schema Version: 5.30.0
    Database: postgres

  2. User A text in the channel X

Expected behavior
User B in channel X always gets a notification on the mobile app.
(no mute)

Observed behavior
The mobile app (ios) does not get notifications sometimes.

Log

  • notification log:
{"level":"error","ts":1651019827.0948913,"caller":"app/notification_push.go:106","msg":"Notification error","logSource":"notifications","ackId":"suwiwafk7jr8pyjwxojdyz8u3r","type":"message","userId":"ojdukremsjgnbbduk5uodnha1w","postId":"6chphxr367g6dnapu5xqpsud3o","channelId":"fs9x5bgnkfd5bkkgk3bcj6pgtr","deviceId":"b3716ca1306b061d5b9d09c96f9e66e49cd80aa7f0a8e9e4577894d81d66de25","status":"Post \"https://push-test.mattermost.com/api/v1/send_push\\": dial tcp: i/o timeout"}
  • network test(curl) in mattermost_app_1 container:
/mattermost # curl -v https://push-test.mattermost.com/api/v1/send_push
*   Trying 52.207.18.93:443...
* TCP_NODELAY set
* Connected to push-test.mattermost.com (52.207.18.93) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.mattermost.com
*  start date: Dec  3 00:00:00 2021 GMT
*  expire date: Dec 30 23:59:59 2022 GMT
*  subjectAltName: host "push-test.mattermost.com" matched cert's "*.mattermost.com"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55b26bcb3580)
> GET /api/v1/send_push HTTP/2
> Host: push-test.mattermost.com
> User-Agent: curl/7.66.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 405 
< date: Wed, 27 Apr 2022 07:09:54 GMT
< content-length: 0
< x-ratelimit-limit: 300
< x-ratelimit-remaining: 299
< x-ratelimit-reset: 1
< 
* Connection #0 to host push-test.mattermost.com left intact
/mattermost # curl -l https://push-test.mattermost.com/api/v1/send_push
/mattermost # curl -v https://push-test.mattermost.com/api/v1/send_push
*   Trying 3.224.61.240:443...
* TCP_NODELAY set
* Connected to push-test.mattermost.com (3.224.61.240) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.mattermost.com
*  start date: Dec  3 00:00:00 2021 GMT
*  expire date: Dec 30 23:59:59 2022 GMT
*  subjectAltName: host "push-test.mattermost.com" matched cert's "*.mattermost.com"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x56182a93a580)
> GET /api/v1/send_push HTTP/2
> Host: push-test.mattermost.com
> User-Agent: curl/7.66.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 405 
< date: Wed, 27 Apr 2022 07:10:08 GMT
< content-length: 0
< x-ratelimit-limit: 300
< x-ratelimit-remaining: 299
< x-ratelimit-reset: 1
< 
* Connection #0 to host push-test.mattermost.com left intact
/mattermost #
2

Hello,

Thanks for reporting your issue. The first issue I noticed is that you’re using a Docker method that we don’t officially support anymore. You can find the latest Docker installation guide here.

I’m not entirely sure why you might be having issues with TPNS other than the fact that it’s not intended for production use. If you need more reliable push notifications, you might want to look into the HPNS service.

3

Hi @evasgit - It looks like sometime the application is failing to connect to our test push notification server. Without further info, it’s hard to really say why. But as @BenLloydPearson pointed out above, we recomend using the HPNS (push.mattermost.com) to run in production environments.

4

How to use HPNS, is it for free? need to upgrade our self-hosted server?

5

You need to have a valid license to use it.