Export (csv): Too many files open

(Edit: since I am a new user, I could NOT upload the requested screenshots. Nice…)

Summary
While exporting chats with a timestamp of 3 days ago, an error occurs which terminates the export.

Error: CsvExport: Unable to write the csv file., unable to open the file data/export/1662588001-1662923819764/csv_export.zip to write the data: open data/export/1662588001-1662923819764/csv_export.zip: too many open files

I have increased the open-files-limit of Ubuntu 20.04, but that not seem to help.

Steps to reproduce

  1. root@myservername:/opt/mattermost# bin/mattermost export csv --exportFrom=1662588001

Expected behavior
I expect the export to finish successfully.

Observed behavior
I observed the following error (this was supposed to be a screenshot):

...
{"timestamp":"2022-09-11 21:17:53.348 +02:00","level":"info","msg":"Stopping Server...","caller":"app/server.go:968"}
{"timestamp":"2022-09-11 21:17:53.349 +02:00","level":"info","msg":"stopping websocket hub connections","caller":"app/web_hub.go:113"}
{"timestamp":"2022-09-11 21:17:53.357 +02:00","level":"info","msg":"Server stopped","caller":"app/server.go:1048"}
Error: CsvExport: Unable to write the csv file., unable to open the file data/export/1662588001-1662923819764/csv_export.zip to write the data: open data/export/1662588001-1662923819764/csv_export.zip: too many open files
Usage:
  mattermost export csv [flags]

Examples:
export csv --exportFrom=12345

Flags:
      --exportFrom int   The timestamp of the earliest post to export, expressed in seconds since the unix epoch. (default -1)
  -h, --help             help for csv

Global Flags:
  -c, --config string   Configuration file to use.

root@myservername:/opt/mattermost#

And after the export fails, the export directory stays empty (this was supposed to be a screenshot):

root@myservername:/opt/mattermost# ll data/export/1662588001-1662923819764/
total 8
drwxr-x--- 2 root root 4096 Sep 11 21:17 ./
drwxr-x--- 3 root root 4096 Sep 11 21:17 ../
root@myservername:/opt/mattermost#

Hi crypt0jan,

what Mattermost server version are you running?
I assume the process is running for some time, so while it is running, could you verify that the increased open files limit has been applied to this process? When it’s running, grep its PID and run cat /proc/<PID>/limits as well as lsof -p <PID> | wc -l to compare the values.
Maybe this one-liner helps with that:

echo "PID    : Files-used/Soft-limit/Hard-Limit"; for pid in $(pidof mattermost); do printf "%-7d: %-10d/%-10d/%-10d\n" $pid $(lsof -p $pid | wc -l) $(awk "-F " '/^Max open files/ { print $4" "$5 }' /proc/$pid/limits); done