Data is automatically deleted?


I have set up a Mattermost server on my Raspberry Pi using Ubuntu server (amd64). I made it accessible via internet using the NoIP DynDNS service and activated TLS using that Let’s Encrypt option.

It now appears that any files posted in a channel are disappearing from the channel after a few days. And by disappearing I mean that they are still shown in the channel, but videos can’t be downloaded (a 404 error appears when playing them and that the file can not be found) and also pictures are shown with preview but only as “error preview” where again the original file can not be found.

When I go to the /opt/mattermost/data folder on my Raspberry Pi I see that the folder where the video and picture was located is deleted. The video link is shown in the 404 error, but the folder is not present anymore on my Raspberry Pi.

What could be the reason for that? It appears that files are being deleted after roughly one week.

We don’t have any plugins, and they are disabled.

Kind regards,

Hi quaderin,

I’m not aware of a feature inside the Mattermost server that would purge user data after a certain amount of time, at least not in the free editions and since you’re running Mattermost on a Raspberry Pi I assume you’re not using the Enterprise edition, right? With the paid subscriptions you can get a feature called Data Retention Policies which is capable of doing something like that, but on the free editions, there’s nothing like that available.

Does the database for your Mattermost server reside on the same filesytsem as the storage? Since you said that the posts in the database are still visible but the links to the attachments show a 404 I think we can rule out filesystem snapshots or corruption if the database as well as the files are residing on the same filesystem.
Speaking of which, can you login to your database and check for the existance of the files in the fileinfo (or Fileinfo if you use MySQL) table?

mm712=# select id,postid,createat,updateat,deleteat,path from fileinfo;
             id             |           postid           |   createat    |   updateat    | deleteat |                                                                                     path
 817un7diubdjfr853zqjyyf4eo | kupqw8d9hprs7qsu5cku5gfqph | 1660276130733 | 1660276130733 |        0 | 20220812/teams/d87r4so6i3nzbgmqnndufui5oy/channels/csjfs3inefywuq3k9qck9p6tnc/users/6jjah7ygkt85be51gfukegfm5a/817un7diubdjfr853zqjyyf4eo/F03TS8P6LE5_WinSCP-5.21.2-Setup.exe
 nsbkmh3pk7n6bp7ufw8rwprz1o | pfthdhxa1tnexy919ny8ro6ich | 1660276097841 | 1660276097841 |        0 | 20220812/teams/d87r4so6i3nzbgmqnndufui5oy/channels/csjfs3inefywuq3k9qck9p6tnc/users/6jjah7ygkt85be51gfukegfm5a/nsbkmh3pk7n6bp7ufw8rwprz1o/F03TALS0D2R_klugscheisser.PNG
(2 rows)

Can you post the record of one of the missing files in your next post?

Also, did you check your system for cronjobs or systemd timers that could be repsonsible for that?

crontab -l
ls /etc/cron.*
ls /var/spool/cron/crontabs
systemctl list-timers
1 Like

Thanks for your answer!

Yes, since I use the free version, I do not have any data retention policy activated.

Entering crontab -l results in “No crontab for ubuntu”.

With ls /etc/cron.* I get:

apport  apt-compat  dpkg  logrotate  man-db




The list from the timers is this one:

Mon 2022-09-26 12:21:45 UTC 23min left    Sun 2022-09-25 23:49:47 UTC 12h ago      fwupd-refresh.timer            fwupd-refresh.service
Mon 2022-09-26 14:29:29 UTC 2h 31min left Mon 2022-09-26 11:22:58 UTC 34min ago    motd-news.timer                motd-news.service
Mon 2022-09-26 14:47:53 UTC 2h 49min left Mon 2022-09-26 07:58:58 UTC 3h 58min ago ua-timer.timer                 ua-timer.service
Mon 2022-09-26 19:55:48 UTC 7h left       Sun 2022-09-25 19:55:48 UTC 16h ago      update-notifier-download.timer update-notifier-download.service
Mon 2022-09-26 20:05:02 UTC 8h left       Sun 2022-09-25 20:05:02 UTC 15h ago      systemd-tmpfiles-clean.timer   systemd-tmpfiles-clean.service
Mon 2022-09-26 20:50:53 UTC 8h left       Mon 2022-09-26 11:27:58 UTC 29min ago    certbot.timer                  certbot.service
Tue 2022-09-27 00:00:00 UTC 12h left      Mon 2022-09-26 00:00:36 UTC 11h ago      dpkg-db-backup.timer           dpkg-db-backup.service
Tue 2022-09-27 00:00:00 UTC 12h left      Mon 2022-09-26 00:00:36 UTC 11h ago      logrotate.timer                logrotate.service
Tue 2022-09-27 00:51:49 UTC 12h left      Mon 2022-09-26 09:09:41 UTC 2h 48min ago apt-daily.timer                apt-daily.service
Tue 2022-09-27 05:08:34 UTC 17h left      Mon 2022-09-26 11:38:58 UTC 18min ago    man-db.timer                   man-db.service
Tue 2022-09-27 06:19:16 UTC 18h left      Mon 2022-09-26 06:05:34 UTC 5h 52min ago apt-daily-upgrade.timer        apt-daily-upgrade.service
Fri 2022-09-30 07:59:33 UTC 3 days left   Tue 2022-09-20 09:14:36 UTC 6 days ago   update-notifier-motd.timer     update-notifier-motd.service
Sun 2022-10-02 03:10:48 UTC 5 days left   Sun 2022-09-25 03:10:52 UTC 1 day 8h ago e2scrub_all.timer              e2scrub_all.service
Mon 2022-10-03 00:01:32 UTC 6 days left   Mon 2022-09-26 00:07:58 UTC 11h ago      fstrim.timer                 

So the only thing that looks like it could do something on a weekly basis is that man-db task? I dont know what it is, maybe something mariadb related? But anyway, this shouldnt delete files in the data folder?

The MariaDB query you proposed resulted in the following output:

MariaDB [mattermost]> select id,postid,createat,updateat,deleteat,path from FileInfo;
| id                         | postid                     | createat      | updateat      | deleteat | path                                                                                                                                                                                    |
| 4kzsgq3b5tnf8r9q7qp14ie4oc | uw9dfoi9ftf69xjkk8di3gme5y | 1663674542951 | 1663674542951 |        0 | 20220920/teams/noteam/channels/ceuh6kyei3fzpqpm7pwhj6618h/users/w9t3rk986id87ns39f44cbi85h/4kzsgq3b5tnf8r9q7qp14ie4oc/rn_image_picker_lib_temp_8b243b38-460e-4bec-81ef-a6acf0a23507.jpg |
| qj6aobuq87ycpm6o5b9bzafgtc | pzjaoqgsfiyyzq4asefqt9crio | 1663534016255 | 1663534016255 |        0 | 20220918/teams/noteam/channels/gkbcecd1ybrdiddfi8zcyoz55o/users/w9t3rk986id87ns39f44cbi85h/qj6aobuq87ycpm6o5b9bzafgtc/20220918_224626.mp4                                               |
| xdzgtjbk7jbmfxgu7ab3ednj5w | eup8689sepy5t8ufcr9wtxtddh | 1663532067221 | 1663532067221 |        0 | 20220918/teams/noteam/channels/gkbcecd1ybrdiddfi8zcyoz55o/users/5fuiab9gjtb7xx44zq4d5sxzrc/xdzgtjbk7jbmfxgu7ab3ednj5w/1.jpg                                                             |
| zx8ggdzmit8rdduwugouyudyzy | ddd9ruz1a3gxid76rqha3q81qy | 1663617821958 | 1663617821958 |        0 | 20220919/teams/noteam/channels/gkbcecd1ybrdiddfi8zcyoz55o/users/w9t3rk986id87ns39f44cbi85h/zx8ggdzmit8rdduwugouyudyzy/rn_image_picker_lib_temp_0e04f20c-ac15-4616-8c3b-050b65b80942.jpg |

So the file info is present in the database, but the files are not present in the data folder.

Basically the mp4 and the jpg files are missing.

I also activated the TLS security after the jpg and the mp4 have been uploaded. I don’t think there is any data deletion when TLS is activated but just in case that that would be important.

I will also uploaded some pictures and videos again to see if it happens again. The video was just a small shot ("gif style, 50 mb), so it also was not too large for raspberry pi (it still has 190gb of free space).

man is the linux utility for opening manual pages and the man-db cronjob updates them, so this should be unrelated.
Did you use the docker deployment and maybe forgot to persist the data volume? This would explain why the database is up2date (the persistence setting seems to work there) and why the data folder is not, but besides that I’m out of ideas what would cause that.

Is your data directory completely empty then and starts at a given date now or do you see older, empty folders? As you can see, Mattermost creates a directory for each day it stores an attachment. If the folders are gone too, there’s definitely something that deletes them then and this is strange.
Please run the following commands and share the output:

ls -lhd /opt/mattermost/data
ls -lh /opt/mattermost/data
df -h 
docker ps
1 Like

Thanks for the reply!

I don’t use any docker containers.

The output is:

drwxrwxr-x 6 mattermost mattermost 4.0K Sep 20 11:49 /opt/mattermost/data


drwxr-x--- 3 mattermost mattermost 4.0K Sep 19 20:03 20220919
drwxr-x--- 3 mattermost mattermost 4.0K Sep 20 11:49 20220920
drwxr-x--- 2 mattermost mattermost 4.0K Sep 20 11:33 plugins
drwxr-x--- 5 mattermost mattermost 4.0K Sep 20 11:46 users


tmpfs           781M  3.1M  778M   1% /run
/dev/sda2       230G  4.4G  216G   2% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sda1       253M  148M  105M  59% /boot/firmware
tmpfs           781M  4.0K  781M   1% /run/user/1000


sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=3941020k,nr_inodes=985255,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=799428k,mode=755,inode64)
/dev/sda2 on / type ext4 (rw,relatime,discard,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14968)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
none on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/var/lib/snapd/snaps/core20_1590.snap on /snap/core20/1590 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/lxd_22927.snap on /snap/lxd/22927 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/core20_1627.snap on /snap/core20/1627 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/lxd_23545.snap on /snap/lxd/23545 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/snapd_16782.snap on /snap/snapd/16782 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/dev/sda1 on /boot/firmware type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,nodev,noexec,relatime,size=799428k,mode=755,inode64)
nsfs on /run/snapd/ns/lxd.mnt type nsfs (rw)
/var/lib/snapd/snaps/snapd_17032.snap on /snap/snapd/17032 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=799424k,nr_inodes=199856,mode=700,uid=1000,gid=1000,inode64)

docker is not installed, therefore there is no docker command.

It’s really strange, does the Sep. 19 dates mean that the rights for mattermost to do anything at all were given after the 19th of september? I mean, I reinstalled the server once by deleting this raspberry pi debian and installing ubuntu. But for that I formatted the usb stick and installed ubuntu freshly, then only installed mattermost.

If for some reasons, these files where send to the “old” system (before the ubuntu installation), then everything, including the mariadb and its knowledge about the files must have been deleted when I installed ubuntu on the usb stick.

I also see that the newest file is actually not being deleted. So maybe the maria mysql database is actually not located on the usb stick?? Or not in a part that gets deleted when you install ubuntu on the usb stick?

I used these tools etc to install ubuntu server amd64 on my raspberry pi usb stick:

So is it possible that the maria db “survived” somehow the transition from the old os to the ubuntu os installation? I dont remember anymore when the missing files were posted (if it was before or after the update to ubuntu…) but my only explaination for the missing files is that they were postet before the ubuntu installation but somehow the maria db survived the ubuntu installation on the usb stick using the Balena Etcher tool…

As you can see, the oldest file here in your /opt/mattermost/data directory is from September 19th, so everything before that is gone and I’m not sure if we will ever find out why this has happened.
Permissions and the rest of the command outputs looks good, nothing suspicious here in my opinion.
The device /dev/sda you have here has only two partitions as it seems, but to verify, please post the output of:

cat /proc/partitions
ls -lh /
find / -type d -name mattermost

If you reinstalled the system without formatting the device (not sure if that’s even possible), you would have some additional folders here for the old installation or an additional partition which is not
mounted as of yet, but we should find that out using the above commands.

1 Like

Thanks again for the reply!

Yes, that’s also what I thought… Could be that some files were already there when I reformatted the stick, but then it’s strange that the info is still in the database.

Output in total:

ubuntu@ubuntu:~$ lsblk
loop0    7:0    0  57.9M  1 loop /snap/core20/1590
loop1    7:1    0    59M  1 loop /snap/core20/1627
loop2    7:2    0  71.8M  1 loop /snap/lxd/22927
loop3    7:3    0   102M  1 loop /snap/lxd/23545
loop5    7:5    0  41.5M  1 loop /snap/snapd/16782
loop6    7:6    0  41.5M  1 loop /snap/snapd/17032
sda      8:0    1 233.3G  0 disk
├─sda1   8:1    1   256M  0 part /boot/firmware
└─sda2   8:2    1   233G  0 part /
ubuntu@ubuntu:~$ cat /proc/partitions
major minor  #blocks  name

   7        0      59248 loop0
   7        1      60432 loop1
   7        2      73540 loop2
   7        3     104484 loop3
   7        5      42524 loop5
   7        6      42520 loop6
   8        0  244580352 sda
   8        1     262144 sda1
   8        2  244317167 sda2
ubuntu@ubuntu:~$ ls -lh /
total 72K
lrwxrwxrwx   1 root root    7 Aug  9 11:54 bin -> usr/bin
drwxr-xr-x   4 root root 4.0K Sep 23 07:01 boot
drwxr-x---   4 root root 4.0K Sep 18 20:42 data
drwxr-xr-x  17 root root  14K Sep 19 19:47 dev
drwxr-xr-x 103 root root 4.0K Sep 23 07:00 etc
drwxr-xr-x   3 root root 4.0K Aug  9 12:05 home
lrwxrwxrwx   1 root root    7 Aug  9 11:54 lib -> usr/lib
drwx------   2 root root  16K Aug  9 12:05 lost+found
drwxr-xr-x   2 root root 4.0K Aug  9 11:54 media
drwxr-xr-x   2 root root 4.0K Aug  9 11:54 mnt
drwxr-xr-x   3 root root 4.0K Sep 18 08:03 opt
drwxr--r--   2 root root 4.0K Sep 18 20:06 plugins
dr-xr-xr-x 204 root root    0 Jan  1  1970 proc
drwx------   5 root root 4.0K Sep 26 12:05 root
drwxr-xr-x  34 root root 1020 Oct  2 08:21 run
lrwxrwxrwx   1 root root    8 Aug  9 11:54 sbin -> usr/sbin
drwxr-xr-x   6 root root 4.0K Aug  9 12:03 snap
drwxr-xr-x   2 root root 4.0K Aug  9 11:54 srv
dr-xr-xr-x  12 root root    0 Jan  1  1970 sys
drwxrwxrwt  13 root root 4.0K Oct  2 08:20 tmp
drwxr-xr-x  11 root root 4.0K Aug  9 11:54 usr
drwxr-xr-x  13 root root 4.0K Aug  9 12:01 var

Somehow I get “permission denied” with pvdisplay and nothing is returned to the console at all when I type “sudo pvdisplay”. Same with lvdisplay :frowning:

sudo lvdisplay -v gives “No volume groups found”.

ubuntu@ubuntu:~$ sudo find / -type d -name mattermost

Don’t know if it helps without the pvdisplay and lvdisplay outputs, unfortunately I am not a regular linux user and probably I jsut forgot a simple thing to make these commands work :frowning:

Yes, that’s OK - just wanted to see if you’re using logical volumes and if the system maybe booted off the wrong one, but since you don’t and there’s just one partition, I have no idea what happened to your system.

I can confirm, though, that Mattermost does not by itself delete data from the filesystem, there’s no functionality available doing that.
Did you experience any further file deletions so far or was this maybe a one-time issue?

1 Like

It was (up to now) a one time issue. Newer files are still there and not being deleted. Very strange and a bit intimidating, but I would just assume for me now that some accident happened in the “learning how to get everything running + reinstalling everything” phase. I never deleted any files myself on purpose, but typically the issue is in front of the computer :). Since everything is running now, I would just keep it up like. Thanks for your awesome support!

Alright, thanks for clarifying. I’ll mark this as “resolved” now :slight_smile:

1 Like