I have set up a Mattermost server on my Raspberry Pi using Ubuntu server (amd64). I made it accessible via internet using the NoIP DynDNS service and activated TLS using that Let’s Encrypt option.
It now appears that any files posted in a channel are disappearing from the channel after a few days. And by disappearing I mean that they are still shown in the channel, but videos can’t be downloaded (a 404 error appears when playing them and that the file can not be found) and also pictures are shown with preview but only as “error preview” where again the original file can not be found.
When I go to the /opt/mattermost/data folder on my Raspberry Pi I see that the folder where the video and picture was located is deleted. The video link is shown in the 404 error, but the folder is not present anymore on my Raspberry Pi.
What could be the reason for that? It appears that files are being deleted after roughly one week.
I’m not aware of a feature inside the Mattermost server that would purge user data after a certain amount of time, at least not in the free editions and since you’re running Mattermost on a Raspberry Pi I assume you’re not using the Enterprise edition, right? With the paid subscriptions you can get a feature called Data Retention Policies which is capable of doing something like that, but on the free editions, there’s nothing like that available.
Does the database for your Mattermost server reside on the same filesytsem as the storage? Since you said that the posts in the database are still visible but the links to the attachments show a 404 I think we can rule out filesystem snapshots or corruption if the database as well as the files are residing on the same filesystem.
Speaking of which, can you login to your database and check for the existance of the files in the fileinfo (or Fileinfo if you use MySQL) table?
Mon 2022-09-26 12:21:45 UTC 23min left Sun 2022-09-25 23:49:47 UTC 12h ago fwupd-refresh.timer fwupd-refresh.service
Mon 2022-09-26 14:29:29 UTC 2h 31min left Mon 2022-09-26 11:22:58 UTC 34min ago motd-news.timer motd-news.service
Mon 2022-09-26 14:47:53 UTC 2h 49min left Mon 2022-09-26 07:58:58 UTC 3h 58min ago ua-timer.timer ua-timer.service
Mon 2022-09-26 19:55:48 UTC 7h left Sun 2022-09-25 19:55:48 UTC 16h ago update-notifier-download.timer update-notifier-download.service
Mon 2022-09-26 20:05:02 UTC 8h left Sun 2022-09-25 20:05:02 UTC 15h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Mon 2022-09-26 20:50:53 UTC 8h left Mon 2022-09-26 11:27:58 UTC 29min ago certbot.timer certbot.service
Tue 2022-09-27 00:00:00 UTC 12h left Mon 2022-09-26 00:00:36 UTC 11h ago dpkg-db-backup.timer dpkg-db-backup.service
Tue 2022-09-27 00:00:00 UTC 12h left Mon 2022-09-26 00:00:36 UTC 11h ago logrotate.timer logrotate.service
Tue 2022-09-27 00:51:49 UTC 12h left Mon 2022-09-26 09:09:41 UTC 2h 48min ago apt-daily.timer apt-daily.service
Tue 2022-09-27 05:08:34 UTC 17h left Mon 2022-09-26 11:38:58 UTC 18min ago man-db.timer man-db.service
Tue 2022-09-27 06:19:16 UTC 18h left Mon 2022-09-26 06:05:34 UTC 5h 52min ago apt-daily-upgrade.timer apt-daily-upgrade.service
Fri 2022-09-30 07:59:33 UTC 3 days left Tue 2022-09-20 09:14:36 UTC 6 days ago update-notifier-motd.timer update-notifier-motd.service
Sun 2022-10-02 03:10:48 UTC 5 days left Sun 2022-09-25 03:10:52 UTC 1 day 8h ago e2scrub_all.timer e2scrub_all.service
Mon 2022-10-03 00:01:32 UTC 6 days left Mon 2022-09-26 00:07:58 UTC 11h ago fstrim.timer
fstrim.service
So the only thing that looks like it could do something on a weekly basis is that man-db task? I dont know what it is, maybe something mariadb related? But anyway, this shouldnt delete files in the data folder?
The MariaDB query you proposed resulted in the following output:
So the file info is present in the database, but the files are not present in the data folder.
Basically the mp4 and the jpg files are missing.
I also activated the TLS security after the jpg and the mp4 have been uploaded. I don’t think there is any data deletion when TLS is activated but just in case that that would be important.
I will also uploaded some pictures and videos again to see if it happens again. The video was just a small shot ("gif style, 50 mb), so it also was not too large for raspberry pi (it still has 190gb of free space).
man is the linux utility for opening manual pages and the man-db cronjob updates them, so this should be unrelated.
Did you use the docker deployment and maybe forgot to persist the data volume? This would explain why the database is up2date (the persistence setting seems to work there) and why the data folder is not, but besides that I’m out of ideas what would cause that.
Is your data directory completely empty then and starts at a given date now or do you see older, empty folders? As you can see, Mattermost creates a directory for each day it stores an attachment. If the folders are gone too, there’s definitely something that deletes them then and this is strange.
Please run the following commands and share the output:
ls -lhd /opt/mattermost/data
ls -lh /opt/mattermost/data
df -h
mount
docker ps
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=3941020k,nr_inodes=985255,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=799428k,mode=755,inode64)
/dev/sda2 on / type ext4 (rw,relatime,discard,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14968)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
none on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/var/lib/snapd/snaps/core20_1590.snap on /snap/core20/1590 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/lxd_22927.snap on /snap/lxd/22927 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/core20_1627.snap on /snap/core20/1627 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/lxd_23545.snap on /snap/lxd/23545 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/snapd_16782.snap on /snap/snapd/16782 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/dev/sda1 on /boot/firmware type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,nodev,noexec,relatime,size=799428k,mode=755,inode64)
nsfs on /run/snapd/ns/lxd.mnt type nsfs (rw)
/var/lib/snapd/snaps/snapd_17032.snap on /snap/snapd/17032 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=799424k,nr_inodes=199856,mode=700,uid=1000,gid=1000,inode64)
docker is not installed, therefore there is no docker command.
It’s really strange, does the Sep. 19 dates mean that the rights for mattermost to do anything at all were given after the 19th of september? I mean, I reinstalled the server once by deleting this raspberry pi debian and installing ubuntu. But for that I formatted the usb stick and installed ubuntu freshly, then only installed mattermost.
If for some reasons, these files where send to the “old” system (before the ubuntu installation), then everything, including the mariadb and its knowledge about the files must have been deleted when I installed ubuntu on the usb stick.
I also see that the newest file is actually not being deleted. So maybe the maria mysql database is actually not located on the usb stick?? Or not in a part that gets deleted when you install ubuntu on the usb stick?
I used these tools etc to install ubuntu server amd64 on my raspberry pi usb stick:
So is it possible that the maria db “survived” somehow the transition from the old os to the ubuntu os installation? I dont remember anymore when the missing files were posted (if it was before or after the update to ubuntu…) but my only explaination for the missing files is that they were postet before the ubuntu installation but somehow the maria db survived the ubuntu installation on the usb stick using the Balena Etcher tool…
As you can see, the oldest file here in your /opt/mattermost/data directory is from September 19th, so everything before that is gone and I’m not sure if we will ever find out why this has happened.
Permissions and the rest of the command outputs looks good, nothing suspicious here in my opinion.
The device /dev/sda you have here has only two partitions as it seems, but to verify, please post the output of:
lsblk
cat /proc/partitions
ls -lh /
pvdisplay
lvdisplay
find / -type d -name mattermost
If you reinstalled the system without formatting the device (not sure if that’s even possible), you would have some additional folders here for the old installation or an additional partition which is not
mounted as of yet, but we should find that out using the above commands.
Yes, that’s also what I thought… Could be that some files were already there when I reformatted the stick, but then it’s strange that the info is still in the database.
Somehow I get “permission denied” with pvdisplay and nothing is returned to the console at all when I type “sudo pvdisplay”. Same with lvdisplay
sudo lvdisplay -v gives “No volume groups found”.
ubuntu@ubuntu:~$ sudo find / -type d -name mattermost
/opt/mattermost
/var/lib/mysql/mattermost
Don’t know if it helps without the pvdisplay and lvdisplay outputs, unfortunately I am not a regular linux user and probably I jsut forgot a simple thing to make these commands work
Yes, that’s OK - just wanted to see if you’re using logical volumes and if the system maybe booted off the wrong one, but since you don’t and there’s just one partition, I have no idea what happened to your system.
I can confirm, though, that Mattermost does not by itself delete data from the filesystem, there’s no functionality available doing that.
Did you experience any further file deletions so far or was this maybe a one-time issue?
It was (up to now) a one time issue. Newer files are still there and not being deleted. Very strange and a bit intimidating, but I would just assume for me now that some accident happened in the “learning how to get everything running + reinstalling everything” phase. I never deleted any files myself on purpose, but typically the issue is in front of the computer :). Since everything is running now, I would just keep it up like. Thanks for your awesome support!
