Push notification (TNPS) not working behind corporate proxy

Troubleshooting
1

For feature requests, please see: http://www.mattermost.org/feature-requests/.

For troubleshooting questions, please post in the following format:

Summary

Somewhere around june our push notification for android stopped working (on iOS it still works i don’t know how). Mattermost log is full of these messages:

|2018-11-08T10:26:45.383+0100|error|app/notification_push.go:188|Device push reported as error for UserId=6d73qi951784ffn19sqshnjfoc SessionId=cj7tjajiypr7dcxqhj53miq3ba message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp 54.175.100.59:80: connect: connection refused|{"user_id": "6d73qi951784ffn19sqshnjfoc"}|
|---|---|---|---|---|
|2018-11-08T10:26:45.383+0100|error|app/notification_push.go:188|Device push reported as error for UserId=6d73qi951784ffn19sqshnjfoc SessionId=cj7tjajiypr7dcxqhj53miq3ba message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp 54.175.100.59:80: connect: connection refused|{"user_id": "6d73qi951784ffn19sqshnjfoc"}|
|2018-11-08T10:26:45.383+0100|error|app/notification_push.go:188|Device push reported as error for UserId=6d73qi951784ffn19sqshnjfoc SessionId=cj7tjajiypr7dcxqhj53miq3ba message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp 54.175.100.59:80: connect: connection refused|{"user_id": "6d73qi951784ffn19sqshnjfoc"}|
|2018-11-08T10:26:45.384+0100|error|app/notification_push.go:188|Device push reported as error for UserId=6d73qi951784ffn19sqshnjfoc SessionId=cj7tjajiypr7dcxqhj53miq3ba message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp 54.175.100.59:80: connect: connection refused|{"user_id": "6d73qi951784ffn19sqshnjfoc"}|
|2018-11-08T10:26:45.384+0100|error|app/notification_push.go:188|Device push reported as error for UserId=6d73qi951784ffn19sqshnjfoc SessionId=cj7tjajiypr7dcxqhj53miq3ba message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp 54.175.100.59:80: connect: connection refused|{"user_id": "6d73qi951784ffn19sqshnjfoc"}|
|2018-11-08T10:26:45.384+0100|error|app/notification_push.go:188|Device push reported as error for UserId=6d73qi951784ffn19sqshnjfoc SessionId=cj7tjajiypr7dcxqhj53miq3ba message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp 54.175.100.59:80: connect: connection refused|{"user_id": "6d73qi951784ffn19sqshnjfoc"}|
|2018-11-08T10:26:45.384+0100|error|app/notification_push.go:188|Device push reported as error for UserId=6d73qi951784ffn19sqshnjfoc SessionId=cj7tjajiypr7dcxqhj53miq3ba message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp 54.175.100.59:80: connect: connection refused|{"user_id": "6d73qi951784ffn19sqshnjfoc"}|
|2018-11-08T10:26:45.384+0100|error|app/notification_push.go:188|Device push reported as error for UserId=6d73qi951784ffn19sqshnjfoc SessionId=cj7tjajiypr7dcxqhj53miq3ba message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp 54.175.100.59:80: connect: connection refused|{"user_id": "6d73qi951784ffn19sqshnjfoc"}|
|2018-11-08T10:26:45.384+0100|error|app/notification_push.go:188|Device push reported as error for UserId=6d73qi951784ffn19sqshnjfoc SessionId=cj7tjajiypr7dcxqhj53miq3ba message=Post http://push-test.mattermost.com/api/v1/send_push: dial tcp 54.175.100.59:80: connect: connection refused|{"user_id": "6d73qi951784ffn19sqshnjfoc"}|

Yet on the corporate proxy there is no log of connection to the push-test.mattermost.com. Seems like push notifications are ignoring http_proxy, https_proxy and no_proxy variables.

I already look at this [SOLVED] How to set HTTP_PROXY variable in order to use external proxy?

and this https://docs.mattermost.com/administration/config-settings.html#allow-untrusted-internal-connections-to

and this Push Notifications not working using TPNS but nothing seems to work.

Steps to reproduce

How can we reproduce the issue (what version are you using?)

we use Mattermost 5.3.1 through GitLab Omnibus behind corporate proxy - every connection outside must go through this proxy.

Expected behavior

To receive push notifications on android devices

Observed behavior

No one receives push notifications.

2

Hi @ElNovi,

Here are some tips / questions:

  1. Verify that the DNS server on the Mattermost host can resolve external domains by adding appropriate forwarding DNS servers, or configure the host to use a different DNS server than 127.0.0.1.

  2. Can you confirm that you’re using the latest version of the Mattermost push proxy server?

  3. Can you check the push notification server logs? I’m not sure where they’re located in your environment, but the default location is in a logs/ directory in the same working directory as the service.

  4. Can you use the command line to log you out of all your sessions? To do this you first run the deactivate command, which removes the sessions, and then you run the activate command to re-activate the user.

3

Hi @amy.blais,

  1. Yes i can reach everything
  2. Well im using your server http://push-test.mattermost.com
  3. Already post them in first post.
  4. I don’t think im able to use command line tool inside GitLab Omnibus
4

Hi @ElNovi,

Letting you know I’m still working on investigating this,

5

@ElNovi Can you help with more details on your config settings for push notifications?

6

Hi @amy.blais,

of course, what info do you need exactly?

7

Hi @ElNovi, these settings will work: https://docs.mattermost.com/administration/config-settings.html#mobile-push.

8

Hi @amy.blais unfortunatly these settings doesn’t work. I already had the right configuration before when it worked. Nothing was changed but it stopped working. I really believe it’s because push notification are ignoring enviroment variables. This was fixed way back with merge request https://github.com/mattermost/mattermost-server/pull/6503. But if i take a look now in the source code i don’t see any trace of this added support. Maybe it was removed?

9

Hi @ElNovi,

I asked our team about this and one of our community members was wondering if you’d be open to sharing your current configuration (Mattermost and system) for everything you’ve touched trying to make this work.

Also, for 5.3.1, http.ProxyFromEnvironment is used: app/notification_push.go:188 is part of sendToPushProxy which had called a.HTTPClient(...) ( app/app.go ) which uses NewHTTPClient ( utils/httpclient.go ).)

10

Hi @amy.blais,

i will send config file to you in private message on https://pre-release.mattermost.com.

But unfortunately i work for different company now and i have no longer access to that server.