I’ve installed Mattermost on a GKE cluster (Google Cloud Kubernetes).
I set up the bot with hubot-matteruser, in the same cluster.
The bot receives some messages, but not all from Mattermost.
And sometimes the bot responds with a message (showing the following logs) that never appears on Mattermost.
[Mon Jan 11 2021 00:59:11 GMT+0000 (Coordinated Universal Time)] DEBUG POST /posts
[Mon Jan 11 2021 00:59:11 GMT+0000 (Coordinated Universal Time)] INFO api url:http://kaggle-mattermost-nodeport/api/v4/posts
[Mon Jan 11 2021 00:59:11 GMT+0000 (Coordinated Universal Time)] DEBUG Posted message.
I first though the problem was linked to the websocket being terminated by the load balancer, but I confirmed it’s set up properly, and the websocket seems to be long lived on the app.
I tried running the very same bot against a managed instance of Mattermost ( (philmod.cloud.mattermost.com) and it works perfectly in that case (i.e. not missed messages).
Hello, @philmod! I’m sorry to hear you’re having difficulties, I’m confident we can get this resolved!
If you enable your Mattermost Instance’s logging level to DEBUG and attempt to reproduce the undesired behavior, do you see anything out of the ordinary in the logs?
In addition, can I ask if you are using a reverse proxy, and if so, which one you are using?
I actually realized this morning there seems to be 2 (related?) problems:
The bot doesn’t receive all the messages sent to it
Some messages sent from the bot (or any user) don’t appear on the Mattermost UI (until the page is refreshed)
The following logs on Mattermost are for the first problem (no logs on the bot side):
{"level":"debug","ts":1610381956.750951,"caller":"web/handlers.go:58","msg":"POST - /api/v4/posts"}
{"level":"warn","ts":1610381956.7685251,"caller":"app/notification.go:333","msg":"Notification not sent","logSource":"notifications","ackId":"","type":"message","userId":"nayb48qybbbw9jg3e864fj8sih","postId":"uaqa5qtzhby8fx5onfmcqdzcco","status":"Not Sent due to preferences"}
For the second problem, I do see a log "msg":"POST - /api/v4/posts" proving a message was sent to Mattermost, but the UI doesn’t show it. Websocket issue?
I think the second problem happens when the user is connected (websocket) to one pod, and the message is sent to the API on a different pod. Is that a known issue?
The only Action Item may be to update the documentation[0] to make sure people understand it won’t work as expected with >1 pod (which is the default configuration).
