Session has expired multi Mattermost server

Hello,
I deployed 2 servers mattermost (version https://releases.mattermost.com/5.39.0/mattermost-5.39.0-linux-amd64.tar.gz) on 2 servers local, with IP addresses are 192.168.0.2 and 192.168.0.3 behind the firewall.
[client] ----->173.16.1.1:8065 (NAT Port TCP 8065 to 8065) -------> 192.168.0.2:8065
[client] ----->173.16.1.1:8064 (NAT Port TCP 8064 to 8064) -------> 192.168.0.3:8064
Mattermost in server 192.168.0.3, I changed listen port in config.json is 8064, and check it is ready to listen on this port.
But: When I login in mattermost version desktop window (4.7.2) or chrome, if I login two servers same time, user auto logout and warning occur: Your session has expired. Please log in again.
Could you please give me some advice?

Hi there @sunq

Would you mind sharing a sanitized copy of your mattermost.log during when the Your session has expired issue occurred and also some context from the firewall log as well to see are there any blocking there so we can have more information what is happening within Mattermost and also your Firewall?

Hi Abdullah.alhabshi,
Many thanks for your response. I configured log mattermost mode DEBUG. And I found some log in file at server port 8064 when session timeout occurs follow my attachment.
My test case with mattermost desktop:

  • I log in to server port 8064, it OK
  • After that, I log in to port 8065 ==> user port 8064 occur “Your session has expired. Please log in again.” and mattermost.log changed.

I tested log in user with server port 8065 with mattermost desktop and user with server port 8064 with chrome. The result: 2 user login OK and I can send message with no problem.
Could you please give me some advice?

My log when problem occur:

{“level”:“debug”,“ts”:1632145994.8005483,“caller”:“web/handlers.go:107”,“msg”:“Received HTTP request”,“method”:“GET”,“url”:“/api/v4/plugins/webapp”,“request_id”:“t93phd4pwifidmt4tfr3a4tace”,“host”:“173.16.1.2:8064”,“scheme”:“”,“status_code”:“200”}
{“level”:“debug”,“ts”:1632145994.8449593,“caller”:“web/handlers.go:107”,“msg”:“Received HTTP request”,“method”:“GET”,“url”:“/api/v4/users/me/teams/k715f5jgajnfi81qs8x59pikhr/channels”,“request_id”:“mhs9sdtbhjnsjeoeiuf88tqyfc”,“host”:“173.16.1.2:8064”,“scheme”:“”,“status_code”:“304”}
{“level”:“debug”,“ts”:1632145994.8450854,“caller”:“web/handlers.go:107”,“msg”:“Received HTTP request”,“method”:“GET”,“url”:“/api/v4/channels/obpxugqy37bwpgr47axcyrq1pe/posts”,“request_id”:“nbx14th5obngxm3wrnezb11der”,“host”:“173.16.1.2:8064”,“scheme”:“”,“status_code”:“200”}
{“level”:“debug”,“ts”:1632145994.846043,“caller”:“web/handlers.go:107”,“msg”:“Received HTTP request”,“method”:“GET”,“url”:“/api/v4/users/me/teams/unread”,“request_id”:“ebmb81u3e7r69r4nc3qh6mgb5h”,“host”:“173.16.1.2:8064”,“scheme”:“”,“status_code”:“200”}
{“level”:“debug”,“ts”:1632145994.846834,“caller”:“web/handlers.go:107”,“msg”:“Received HTTP request”,“method”:“GET”,“url”:“/api/v4/users/me/teams/k715f5jgajnfi81qs8x59pikhr/channels/members”,“request_id”:“k1rpbr8detyrzedk14r79zkd3r”,“host”:“173.16.1.2:8064”,“scheme”:“”,“status_code”:“200”}
{“level”:“debug”,“ts”:1632145998.8076515,“caller”:“wsapi/websocket_handler.go:25”,“msg”:“Websocket request”,“action”:“user_update_active_status”}
{“level”:“debug”,“ts”:1632146019.6967795,“caller”:“web/handlers.go:107”,“msg”:“Received HTTP request”,“method”:“POST”,“url”:“/api/v4/channels/members/me/view”,“request_id”:“8zaic31hf7fkuk1wfiy3higkmy”,“host”:“173.16.1.1:8064”,“scheme”:“”,“status_code”:“200”}
{“level”:“debug”,“ts”:1632146019.7043798,“caller”:“web/handlers.go:107”,“msg”:“Received HTTP request”,“method”:“GET”,“url”:“/api/v4/users/me/teams/k715f5jgajnfi81qs8x59pikhr/channels”,“request_id”:“5434anq43tghbpyophpu1suweh”,“host”:“173.16.1.1:8064”,“scheme”:“”,“status_code”:“304”}
{“level”:“debug”,“ts”:1632146019.7055552,“caller”:“web/handlers.go:107”,“msg”:“Received HTTP request”,“method”:“GET”,“url”:“/api/v4/users/me/teams/k715f5jgajnfi81qs8x59pikhr/channels/members”,“request_id”:“86up9ekidffptq1riuu16defao”,“host”:“173.16.1.1:8064”,“scheme”:“”,“status_code”:“200”}
{“level”:“debug”,“ts”:1632146019.708069,“caller”:“web/handlers.go:107”,“msg”:“Received HTTP request”,“method”:“POST”,“url”:“/api/v4/channels/members/me/view”,“request_id”:“6noydysxypb17dshocw4kw613w”,“host”:“173.16.1.1:8064”,“scheme”:“”,“status_code”:“200”}
{“level”:“debug”,“ts”:1632146019.7183013,“caller”:“web/handlers.go:107”,“msg”:“Received HTTP request”,“method”:“POST”,“url”:“/api/v4/channels/members/me/view”,“request_id”:“bubu67frf3ns3xzzok67gw95bo”,“host”:“173.16.1.1:8064”,“scheme”:“”,“status_code”:“200”}
{“level”:“debug”,“ts”:1632146020.7829642,“caller”:“web/handlers.go:107”,“msg”:“Received HTTP request”,“method”:“POST”,“url”:“/api/v4/channels/members/me/view”,“request_id”:“jia1gk7bo7gbxf6cbr88m6cwpc”,“host”:“173.16.1.1:8064”,“scheme”:“”,“status_code”:“200”}
{“level”:“debug”,“ts”:1632146024.7912605,“caller”:“web/handlers.go:107”,“msg”:“Received HTTP request”,“method”:“POST”,“url”:“/api/v4/users/status/ids”,“request_id”:“izj1qcxdq78ixrqfhxpxxbbo4r”,“host”:“173.16.1.1:8064”,“scheme”:“”,“status_code”:“200”}
{“level”:“warn”,“ts”:1632146027.715514,“caller”:“app/session.go:89”,“msg”:“Error while creating session for user access token”,“error”:“createSessionForUserAccessToken: Invalid or missing token., resource: UserAccessToken id: token=g59kak3h6fgbmy7n378jd3uwjw”}
{“level”:“info”,“ts”:1632146027.716133,“caller”:“mlog/log.go:237”,“msg”:“Invalid session”,“error”:“GetSession: Invalid session token=g59kak3h6fgbmy7n378jd3uwjw, err=, “}
{“level”:“debug”,“ts”:1632146027.7166164,“caller”:“mlog/log.go:230”,“msg”:“Invalid or expired session, please login again.”,“path”:”/api/v4/channels/members/me/view”,“request_id”:“cg1ojtro63bhz8u3b3wysccxjw”,“ip_addr”:“192.168.0.1”,“user_id”:“”,“method”:“POST”,“err_where”:“ServeHTTP”,“http_code”:401,“err_details”:“token=g59kak3h6fgbmy7n378jd3uwjw”}
{“level”:“debug”,“ts”:1632146027.7170842,“caller”:“web/handlers.go:107”,“msg”:“Received HTTP request”,“method”:“POST”,“url”:“/api/v4/channels/members/me/view”,“request_id”:“cg1ojtro63bhz8u3b3wysccxjw”,“host”:“173.16.1.1:8064”,“scheme”:“”,“status_code”:“401”}
{“level”:“debug”,“ts”:1632146027.7578616,“caller”:“mlog/log.go:230”,“msg”:“Invalid or expired session, please login again.”,“path”:“/api/v4/channels/members/me/view”,“request_id”:“nfpbuu4hfbrg5xum98q5ucwrye”,“ip_addr”:“192.168.0.1”,“user_id”:“”,“method”:“POST”,“err_where”:“”,“http_code”:401,“err_details”:“UserRequired”}
{“level”:“debug”,“ts”:1632146027.7584581,“caller”:“web/handlers.go:107”,“msg”:“Received HTTP request”,“method”:“POST”,“url”:“/api/v4/channels/members/me/view”,“request_id”:“nfpbuu4hfbrg5xum98q5ucwrye”,“host”:“173.16.1.1:8064”,“scheme”:“”,“status_code”:“401”}
{“level”:“debug”,“ts”:1632146027.790572,“caller”:“app/web_conn.go:722”,“msg”:“websocket.read: client side closed socket”,“user_id”:“i9fydazybjrufjj1wjgxfypwro”}
{“level”:“debug”,“ts”:1632146027.7912219,“caller”:“web/handlers.go:107”,“msg”:“Received HTTP request”,“method”:“GET”,“url”:“/api/v4/websocket”,“request_id”:“necfxwr8z7nhbr3xg7rx1c1why”,“host”:“173.16.1.1:8064”,“scheme”:“”}
{“level”:“debug”,“ts”:1632146027.7961652,“caller”:“web/handlers.go:107”,“msg”:“Received HTTP request”,“method”:“GET”,“url”:“/”,“request_id”:“r5xxusfr5jbejjhdqke539r16e”,“host”:“173.16.1.1:8064”,“scheme”:“”,“status_code”:“304”}
{“level”:“debug”,“ts”:1632146028.1217577,“caller”:“web/handlers.go:107”,“msg”:“Received HTTP request”,“method”:“GET”,“url”:“/api/v4/license/client”,“request_id”:“b7bp9a6gbbfk5849msrrt7s79e”,“host”:“173.16.1.1:8064”,“scheme”:“”,“status_code”:“200”}
{“level”:“debug”,“ts”:1632146028.1221268,“caller”:“web/handlers.go:107”,“msg”:“Received HTTP request”,“method”:“GET”,“url”:“/api/v4/config/client”,“request_id”:“ua65oeika7b1pfiyzmzquduhkh”,“host”:“173.16.1.1:8064”,“scheme”:“”,“status_code”:“200”}
{“level”:“debug”,“ts”:1632146028.1586268,“caller”:“web/handlers.go:107”,“msg”:“Received HTTP request”,“method”:“GET”,“url”:“/api/v4/plugins/webapp”,“request_id”:“nbgqt877mjfs3nbgotqrchmg1r”,“host”:“173.16.1.1:8064”,“scheme”:“”,“status_code”:“200”}
{“level”:“debug”,“ts”:1632146028.327315,“caller”:“mlog/log.go:230”,“msg”:“You do not have the appropriate permissions.”,“path”:“/api/v4/logs”,“request_id”:“kurhdkuw378mue9f15coco8buw”,“ip_addr”:“192.168.0.1”,“user_id”:“”,“method”:“POST”,“err_where”:“postLog”,“http_code”:403,“err_details”:“”}
{“level”:“debug”,“ts”:1632146028.3279097,“caller”:“web/handlers.go:107”,“msg”:“Received HTTP request”,“method”:“POST”,“url”:“/api/v4/logs”,“request_id”:“kurhdkuw378mue9f15coco8buw”,“host”:“173.16.1.1:8064”,“scheme”:“”,“status_code”:“403”}

Hi sunq,

I just stumbled upon this old topic and saw that it does not have an answer as of yet: The problem is, that your browser and the desktop apps store a session based on the hostname or IP address, not on the port, so if your clients access two different Mattermost installations with the same name or IP address, the session cookies will only be valid for one. You can not log in with two users in the same browser to the same installation and you cannot login to two different Mattermost installations on the same IP but with different ports, so in your case hostnames could help.

You can set up two local hostnames/DNS records for mattermost1 and mattermost2, both pointing to 173.16.1.1 and when you access them using the different hostnames, the cookies will also be stored for these hostnames and you should be good to go.
Please make sure that you also set up the SiteURL in both instances accordingly and nowadays I think it also needs to be a FQDN.
You might also think of setting up an nginx reverse proxy instead of the two NATs on your server with the IP 173.16.1.1 - that way you could set up two fully functioning domain names for the two instances and reverse proxy them to the port 8065 on your two internal nodes.