I get the following error sometimes when typing outgoing webhook keywords into mattermost:
Event POST failed, err=Post http://mywebendpoint.com/somepath/somefile.json: dial tcp: i/o timeout
Steps to reproduce
Type in the webhook keyword. Press enter to submit.
This tends to happen if I haven’t typed the webhook keyword in the last minute or so. Perhaps it’s related to websockets not playing nice? Webserver is apache and I don’t think it’s set up to handle websockets well, just plain http requests.
Expected behavior
Webhook should post a request to my server endpoint any time a keyword is triggered.
Observed behavior
Sometimes typing in a keyword doesn’t post a request to my server. Typically when it’s the first time someone has used the hook in the last minute or so.
If you still have issues after going through the webhook set-up documentation, please let us know which version of Mattermost you have installed and a little more about your installation environment so we can help troubleshoot?
I reviewed that documentation thoroughly while setting up the webhook. I have yet to resolve this issue. We are currently running the mattermost server version 3.10.0 on one server, and the web endpoiont it is attempting to reach is returning json, is on a separate (physical) server, and is being hosted up by an apache web server. The web endpoint seems to be working fine as it responds to curl requests properly.
The interesting part is the mattermost integration works most of the time, the only time it fails to post is upon the first outgoing hook triggering. If you retype the trigger word after the initial failure, it works just fine.
How quickly does your webhook usually respond to the server? I’m not sure what the exact timeout is for those requests, but it might be taking too long to respond.
Also, is it possible to confirm if your webhook is receiving anything from the server in those cases? Like does it have its own logs that might explain more?
The endpoint responds pretty quick to my curl testing. Certainly not long enough to warrant any reasonable timeout. It has it’s own logs and when it fails, it doesn’t even hit my server endpoint, it’s like mattermost tries to post and can’t reach the endpoint server at all on the first try, but then if I try immediately afterward it’s completely fine.
Also, I ran a check on the response time. The endpoint server responds to curl requests made from the mattermost server properly with around 78ms response time.
That response time definitely sounds fine, so that shouldn’t be the issue. You said the web server and Mattermost are on separate machines, so is there anything between them that could be causing the connection to be interrupted, like a firewall or proxy? If so, they might have additional logs that could explain why the connection appears to be dropped
Sorry, I’m not sure what the issue could be. The only other thing I could think is that the endpoint appears to be returning a static file instead of something dynamic, but I can’t see how that would cause it to time out. If it was an https request, it might be a certificate issue that could be fixed by changing the EnableInsecureOutgoingConnections setting, but it looks like you’re using plain http and that also wouldn’t make sense if it was only a periodic issue.
You’re the best! I completely forgot that our endpoint redirects http to https. Switching EnableInsecureOutgoingConnections to true seems to have fixed the issue.