Hello, @richardsonje
That looks a handful. Can you please log out from every session and only log in to the iOS device to reproduce the issue before generating a new notification.log? That will help to narrow down the issue better. Thanks.
Been a little crazy at work and home and just settling down. Ok - should I sign out EVERYONE or just me from all devices? I think I did that not long before I sent that log but I can do this again
OK, here’s the result from the query you requested:
±---------------------------±---------±----------------------±--------------------------------------------------------------------------±-----------------------------------------------------------------------------------------------------------------------------+
| Id | Username | Email | DeviceId | Props |
±---------------------------±---------±----------------------±--------------------------------------------------------------------------±-----------------------------------------------------------------------------------------------------------------------------+
| NULL | NULL | NULL | apple_rn:5570179a889af069398d162868ece9d766419e090e2bd0f8feb5ef7f5e8144f2 | {“browser”:“Safari/13.3”,“csrf”:“pfxanmzbxinkzedx5k6ianm67y”,“is_guest”:“false”,“os”:“iOS”,“platform”:“iPhone”} |
| bsaan4e6pfbrxd8umc973ymxty | jeremy | jeremy@domain.one | apple_rn:9f2844888b6c582ab85cbecb5ce174ba9618b82b710534c1e3a595b54235b8f4 | {“browser”:“Safari/13.3.1”,“csrf”:“pkbdqbga17fzmbtuzsic1nmurh”,“is_guest”:“false”,“os”:“iOS”,“platform”:“iPhone”} |
| NULL | NULL | NULL | | {“browser”:“Safari/13.0.5”,“csrf”:“x6tqesbnopnq5gp74imf7ob51a”,“is_guest”:“false”,“os”:“iOS”,“platform”:“iPhone”} |
| NULL | NULL | NULL | apple_rn:de9ceb1b657d67695f1fd93f886dc9bbb8f40094931bd87643cfd05d29a86adb | {“browser”:“Safari/13.3.1”,“csrf”:“yu5riurj6fy95q3exjb78w9bxa”,“is_guest”:“false”,“os”:“iOS”,“platform”:“iPhone”} |
| NULL | NULL | NULL | | {“browser”:“Chrome/79.0.3945”,“csrf”:“694txrricpnt5xyp1zqd44dxya”,“is_guest”:“false”,“os”:“Android”,“platform”:“Linux”} |
| NULL | NULL | NULL | | {“browser”:“Chrome/80.0.3987”,“csrf”:“1r6ih96yrpbxffiug1sz1kz5aa”,“is_guest”:“false”,“os”:“Windows 10”,“platform”:“Windows”} |
| NULL | NULL | NULL | | {“browser”:“Firefox/74.0”,“csrf”:“8r58opkijpgddj64oqwcdszmeo”,“is_guest”:“false”,“os”:“Mac OS”,“platform”:“Macintosh”} |
±---------------------------±---------±----------------------±--------------------------------------------------------------------------±-----------------------------------------------------------------------------------------------------------------------------+
Here’s my error log:
{“level”:“error”,“ts”:1587400887.3557847,“caller”:“mlog/log.go:175”,“msg”:“Enter a valid email or username and/or password.”,“path”:“/api/v4/users/login”,“request_id”:“5xnbmf5sgfyoxk1a7rpwarqpcr”,“ip_addr”:“10.10.3.162”,“user_id”:“”,“method”:“POST”,“err_where”:“login”,“http_code”:401,“err_details”:“”}
{“level”:“error”,“ts”:1587400931.431108,“caller”:“mlog/log.go:175”,“msg”:“Enter a valid email or username and/or password.”,“path”:“/api/v4/users/login”,“request_id”:“bxcsicbt8tymfbrbereifobppa”,“ip_addr”:“10.10.3.162”,“user_id”:“”,“method”:“POST”,“err_where”:“login”,“http_code”:401,“err_details”:“”}
{“level”:“error”,“ts”:1587401002.8468738,“caller”:“mlog/log.go:175”,“msg”:“Enter a valid email or username and/or password.”,“path”:“/api/v4/users/login”,“request_id”:“swnmirbykfrupydinosorrix9h”,“ip_addr”:“10.10.3.162”,“user_id”:“”,“method”:“POST”,“err_where”:“login”,“http_code”:401,“err_details”:“”}
{“level”:“error”,“ts”:1587401017.8027334,“caller”:“mlog/log.go:175”,“msg”:“Enter a valid email or username and/or password.”,“path”:“/api/v4/users/login”,“request_id”:“kg6ufdmmxtg39g13naw8jxs1fh”,“ip_addr”:“10.10.3.162”,“user_id”:“”,“method”:“POST”,“err_where”:“login”,“http_code”:401,“err_details”:“”}
{“level”:“error”,“ts”:1587401044.8230484,“caller”:“mlog/log.go:175”,“msg”:“Enter a valid email or username and/or password.”,“path”:“/api/v4/users/login”,“request_id”:“opdrxohr5bbumm639c6jjr5sgo”,“ip_addr”:“205.237.121.1”,“user_id”:“”,“method”:“POST”,“err_where”:“login”,“http_code”:401,“err_details”:“”}
{“level”:“error”,“ts”:1587401055.4120007,“caller”:“mlog/log.go:175”,“msg”:“Enter a valid email or username and/or password.”,“path”:“/api/v4/users/login”,“request_id”:“936riusf378bzgmsj9p9u9wqje”,“ip_addr”:“205.237.121.1”,“user_id”:“”,“method”:“POST”,“err_where”:“login”,“http_code”:401,“err_details”:“”}
And I’m also seeing these:
{“level”:“error”,“ts”:1587711497.1088774,“caller”:“mailservice/mail.go:153”,“msg”:“Failed to open a connection to SMTP server”,“error”:“EOF”}
{“level”:“error”,“ts”:1587711497.10896,“caller”:“app/notification_email.go:103”,“msg”:“Error while sending the email”,“user_email”:“jeremy@domain.one”,“error”:“SendMail: Failed to open TLS connection, EOF”}
{“level”:“info”,“ts”:1587228834.84112,“caller”:“mlog/log.go:167”,“msg”:“Invalid session”,“error”:"GetSession: Invalid session token=xyk4yx9xn3rstb3cfcajzdt7wa, err=, "}
{“level”:“info”,“ts”:1587228834.8415837,“caller”:“mlog/log.go:167”,“msg”:“Invalid session”,“error”:"GetSession: Invalid session token=xyk4yx9xn3rstb3cfcajzdt7wa, err=, "}
{“level”:“info”,“ts”:1587228834.88413,“caller”:“mlog/log.go:167”,“msg”:“Invalid session”,“error”:"GetSession: Invalid session token=xyk4yx9xn3rstb3cfcajzdt7wa, err=, "}
{“level”:“info”,“ts”:1587228834.8842106,“caller”:“mlog/log.go:167”,“msg”:“Invalid session”,“error”:"GetSession: Invalid session token=xyk4yx9xn3rstb3cfcajzdt7wa, err=, "}
{“level”:“info”,“ts”:1587228834.8846433,“caller”:“mlog/log.go:167”,“msg”:“Invalid session”,“error”:"GetSession: Invalid session token=xyk4yx9xn3rstb3cfcajzdt7wa, err=, "}
{“level”:“info”,“ts”:1587228834.8848343,“caller”:“mlog/log.go:167”,“msg”:“Invalid session”,“error”:"GetSession: Invalid session token=xyk4yx9xn3rstb3cfcajzdt7wa, err=, "}
{“level”:“info”,“ts”:1587228834.8850775,“caller”:“mlog/log.go:167”,“msg”:“Invalid session”,“error”:"GetSession: Invalid session token=xyk4yx9xn3rstb3cfcajzdt7wa, err=, "}
Hello, @richardsonje
Thanks for the update. So, I believe that we will be focusin on the ID bsaan4e6pfbrxd8umc973ymxty8 assigned to the user jeremy@domain.one:
| bsaan4e6pfbrxd8umc973ymxty | jeremy | jeremy@domain.one | apple_rn:9f2844888b6c582ab85cbecb5ce174ba9618b82b710534c1e3a595b54235b8f4 | {“browser”:“Safari/13.3.1”,“csrf”:“pkbdqbga17fzmbtuzsic1nmurh”,“is_guest”:“false”,“os”:“iOS”,“platform”:“iPhone”} |
The error log you shared only shows the error with the login:
Enter a valid email or username and/or password
However, the last part of the log is the one that catches my attention, the info related to invalid session token. Are you getting that from the mattermost.log or notification.log itself and is the token xyk4yx9xn3rstb3cfcajzdt7wa associated with any of the accounts?
Can you also install the latest 1.30.1 mobile app too? Once done, please go through the Testing push notifications and let me know how it goes.
I got that info from the system console under the logs - is the notifications log on the linux box itself? I can provide that.
I’ll do the push notifications test later tonight
One of these day’s i’ll have free time again and won’t take weeks at a time to stay on this.
I’ve just copied up the logs to here:
I don’t think I have the tokens setup really but I can check if you tell me how. I made sure I have the log level set to DEBUG and then I did the troubleshooting process after I signed out of all devices that were not my iPhone. I get the push notification, but the badge indication never shows up. I am on the latest version of the iOS app.
Anything else i can provide to help?
Ok i decided to test this a couple different ways this morning. I setup a rocket.chat server since they are pretty simple out of curiosity. I found that I have issues push notifications there, too.
So, I shut that down and started testing mm again and noticed something. I forced a logout of all my devices and logged into just my iphone. Then i logged in as another user from my desktop app. I noticed that I only get maybe 1 out of every 5 or 6 push notifications and it’s even more rare to get the badge indicator. I had a friend messaging me at the same time and I’ll get maybe the first push notification and then i’ll get the first push notification from the other user then it kind of takes a break and i get nothing. I didn’t get a badge icon indicator at all.
I am on the latest version of the iOS app. All my users except 1 are iOS users.
In the last post you mention a token - can you elaborate on that? What are the tokens used for and how do I verify this?
I reverted back to the snapshot of my chat VM to right before this upgrade … and now they work just fine. As soon as I upgrade, it breaks the notifications. If I keep it on the older version, works perfect.
@ahmaddanial FYI on the above information.
Hi, @richardsonje
Apologies as this slipped off my radar when you responded earlier. May I know if you are open to raise an official support ticket that will allow me to look into this further? Since this is an edge case that will require me to dig into more information that may be sensitive to public,
By the way, the token that I was referring to earlier was the tokens associated to a user when they are logged in to a session. For example:
mysql> SELECT * FROM Sessions WHERE UserId = "31c9fb8mminuf8hpbq1ossk9iy"\G
*************************** 1. row ***************************
Id: 9u9e6ufcyjrf3jsj9m57rnk8nh
Token: rwog9poo8bg37ptda7npxaj1je
CreateAt: 1590982079889
ExpiresAt: 1606534079889
LastActivityAt: 1590984672774
UserId: 31c9fb8mminuf8hpbq1ossk9iy
DeviceId:
Roles: system_admin system_user
IsOAuth: 0
Props: {"browser":"Chrome/84.0.4147","csrf":"q7hyozaxz3y7fn3o44cpubzj4h","is_guest":"false","os":"Mac OS","platform":"Macintosh"}
*************************** 2. row ***************************
Id: gjagxnh41tb7uck3q335h3u5ma
Token: t84x9be95if6xdqyjfa5f6r6dh
CreateAt: 1591047404307
ExpiresAt: 1606599404307
LastActivityAt: 1591047404307
UserId: 31c9fb8mminuf8hpbq1ossk9iy
DeviceId: apple:eac503aecb977090e0cbabe0f06b88cf80ae3deb782b13778d61704103e1eccc
Roles: system_admin system_user
IsOAuth: 0
Props: {"browser":"Safari/13.5","csrf":"jbmczgc7dpn4dqi5mh3acg1h3w","is_guest":"false","os":"iOS","platform":"iPhone"}
2 rows in set (0.00 sec)
As we can see here, my account is associated with 2 tokens. One which is logged in to the Chrome browser and the other one on the iPhone device.
-
Since you mentioned that you have reverted to the snapshot prior to the upgrade, may I know what is the version that you are currently on?
-
Does this version have the Apache proxy configured for it and is the unread count going through properly if it is in place?
-
Which version did you upgrade to based on the latest statement here?
- When you tested with rocket.chat, did you use the same proxy configuration or were you just using a direct connection?
Yes I’m ok with adding an official support ticket.
5.16.0 is what it’s on now
So it’s going through the nginx reverse proxy without any changes to the config after I did the VM snapshot restore. So, this could “prove” that the proxy isn’t the issue here, possibly.
Looks like i tried upgrading to 5.20.
I still went through a proxy with their recommended config for nginx. I set it up on a separate domain but setup the same concept with a letsencrypt SSL cert.
Hello, @richardsonje
Got it. For the sake of testing, may I know if there are any possibilities of removing the proxy and connect directly to the server to verify that the unread counter works? I have yet to come across issues similar to this since the push notification actually goes through but for some reason, the unread counter does not get updated accordingly on the mobile level.
Yeah I can do that. What ports do I need to forward to the chat server?
Hello, @richardsonje
The ports that you will need to configure would be 80 and 443 based on configuration here. Let me know how it goes. Thanks.
OK, we had a serious outage at work i’m still dealing with but that stole my attention.
I upgraded to 5.25.0 today since I started getting the update warning. I have the same issue. As soon as I upgrade it breaks the notifications. I pointed 80/443 directly to the VM but that just prevents it from working at all. Do I also need to forward 8065?