Messages missing 1 week after upgrading

I upgraded my mattermost server to 7.10.5 last sunday (30 Jul) from 7.9.1 and everyone was able to send messages

This morning (7 Aug) all the messages were missing from 30 Jul.

After some investigation, logs showed that it couldn’t connect to DB for the whole week but was able to run and accept and serve messages. (how’s that possible? it’s super strange!)

Further probing, the postgres database owner has changed to an admin user instead of the usual user stated in my config.json admin user has no rights on the mattermost database.

Is there any way to recover the week’s worth of messages? I am able to see the attachments in the data folder.

Mattermost is running on Apache Linux (QNAP) server on Postgres 10. We have turned off MM server for now.
To add on, one of my colleague’s phone is still able to see the week’s messsages on her Android phone’s MM app. Anyway to sync it from the phone back to the server?

Settled, seems that it was postgres pg_wal issue.

1 Like

Having same issue here @yanuk, can you expand on what solution to pg_wal issue was?
Thanks in advance

it was quite some time ago. I vaguely remember that postgres lost sync with the pg_wal files but lucky for me there there was a backup checkpoint and I restored that checkpoint and got back my week’s worth of data stored in the WAL. Still puzzles me how MM reports dB error but I was still able to use it.

These days you can use chatgpt to help you troubleshoot postgres issues. Just make sure to backup the original data before working on it

Appreciate the reply. In our case it seems the opposite was the case?

Before the server upgrade we converted the DB to postgres, let it run for a week to make sure all was working as expected, all the while MM was not reporting any errors, but after server upgrade no posts after the DB conversion appeared?

We’re still working with MM support to see if there’s any chance of recovering posts from that gap but not looking good.

In any case, again, thanks for the follow up. Very much appreciated.

I hope your issue gets resolved soon!