Summary
No notifications are working, and getting the following error in notifications.log: dial tcp: lookup push-test.mattermost.com: i/o timeout
Steps to reproduce
Using Team Edition 7.8 on AlmaLinux 8.7 with Docker and Nginx over SSL.
Expected behavior
Push notifications have not worked since I deployed the Docker container. TPNS is enabled in console with proper push notification settings set correctly.
Observed behavior
Below is a copy/paste of one of the error messages in the notifications.log. Unfortunately, no other errors can be found in any other logs. I’ve scoured these forums, Google and Github, and even tried GPT and Sydney, but none of the suggestions have worked.
this would indicate that your Mattermost server is unable to connect ot the push proxy on a network level. Do you have outbound ACLs active which would disallow this connection?
Running the following command on your Mattermot server should work and return a simple HTML telling you that this is the pushproxy service:
BTW: This is only the test server, it should not be used for production services. The production service is available at push.mattermost.com or hpns-de.mattermost.com if you want to use one hosted in Germany.
That’s interesting… So when you’re in the docker container (docker exec ...) you can curl the URL without problems, but as soon as the Mattermost application tries to send a notification, it times out? Does this happen for all three (test, prod-us, prod-de) push service servers?
I haven’t tried anything other than the TPNS. I need a subscription to use the others, don’t I?
Here is the output of the command, works fine.
[root@adsl-70-237-101-177 ~]# docker exec 3f17543172e1 curl https://push-test.mattermost.com/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 47 100 47 0 0 11 0 0:00:04 0:00:04 --:--:-- 11
<html><body>Mattermost Push Proxy</body></html>[root@adsl-70-237-101-177 ~]#
There definitely is a problem with the outbound connection of your Mattermost application container, or the internet is so slow on your end, that it times out.
How long does downloading this file from within your container take?
Yeah, not an Internet issue. Dedicated 1GB fiber line.
That download in 4s.
[root@mm ~]# docker exec 86aa48dc9650 curl --output /tmp/temp.tar.gz https://plugins-store.test.mattermost.com/release/mattermost-plugin-playbooks-v1.28.2-linux-amd64.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 18.7M 100 18.7M 0 0 3857k 0 0:00:04 0:00:04 --:--:-- 4662k
I was able to get the playbook and boards working again. Not sure what happened, but I renamed the folders in data, rebooted, and they re-downloaded and started working again.
Push notifications started working earlier today for about 10 mins, then stopped and haven’t worked again since. I’m unsure what happened and why they kicked in and stopped.
I’m a bit puzzled by the messages I can see in your logs, honestly.
The message clearly states that there’s an i/o timeout trying to connect to the external resources, but whenever you run the commands manually, they work.
The internal http method of the Mattermost server seems to have some fixed timeouts set for DNS resolution and content reception after the connection has been established, maybe your DNS resolution is slow inside the container or something like this and therefore you exceed the configured timeouts?
I realized we had an old search domain configured in resolv.conf that is no longer in use. It had to timeout, then process the next in line. Which, of course, caused too much of a delay in DNS resolution, and it timed out.
