Unable to upload files on dockerized MM

Troubleshooting
1

Summary
Uploading files in any of the channels does not complete (both web and client).

Steps to reproduce
I am running MM from a docker image according to instructions found on https://docs.mattermost.com/install/install-docker.html in the Deploy Mattermost on Docker for production use paragraph on a Debian 11 host with my own external NginX. Here is the release:

Enterprise Edition - Free license
Mattermost Version: 8.1.9
Database Schema Version: 113
Build Number: 7670458045
Database: postgres

On the host system shell prompt, here is what I see being the content of the directory:

drwxr-xr-x 9 bob bob  4096 Apr 17 17:23 .
drwxr-xr-x 3 bob bob  4096 Apr  4 16:38 ..
drwxr-xr-x 2 bob bob  4096 Mar 30 10:36 .circleci
drwxr-xr-x 3 bob bob  4096 Mar 30 10:36 contrib
-rw-r--r-- 1 bob bob  1021 Mar 30 10:36 docker-compose.nginx.yml
-rw-r--r-- 1 bob bob   140 Apr 17 17:23 docker-compose.without-nginx.yml
-rw-r--r-- 1 bob bob  2293 Apr 17 17:22 docker-compose.yml
drwxr-xr-x 2 bob bob  4096 Mar 30 10:36 docs
-rw-r--r-- 1 bob bob  4327 Mar 30 11:02 .env
-rw-r--r-- 1 bob bob  4324 Mar 30 10:36 env.example
drwxr-xr-x 8 bob bob  4096 Mar 30 10:36 .git
-rw-r--r-- 1 bob bob    24 Mar 30 10:36 .gitignore
-rw-r--r-- 1 bob bob 11357 Mar 30 10:36 LICENSE
drwxr-xr-x 3 bob bob  4096 Mar 30 10:36 nginx
-rw-r--r-- 1 bob bob   748 Mar 30 10:36 README.md
drwxr-xr-x 2 bob bob  4096 Mar 30 10:36 scripts
drwxr-xr-x 4 bob bob  4096 Mar 30 11:02 volumes

please note that ./volumes is owned by host user bob. Within volumes I have:

bob@newisa:~/mattermost/docker/volumes$ ls -la
total 16
drwxr-xr-x 4 bob bob 4096 Mar 30 11:02 .
drwxr-xr-x 9 bob bob 4096 Apr 17 17:23 ..
drwxr-xr-x 3 bob bob 4096 Mar 30 10:39 app
drwxr-xr-x 3 bob bob 4096 Mar 30 11:02 db

bob@newisa:~/mattermost/docker/volumes/app$ ls -la
total 12
drwxr-xr-x 3 bob  bob  4096 Mar 30 10:39 .
drwxr-xr-x 4 bob  bob  4096 Mar 30 11:02 ..
drwxr-xr-x 8 2000 2000 4096 Mar 30 10:39 mattermost

bob@newisa:~/mattermost/docker/volumes/app/mattermost$ ls -la
total 32
drwxr-xr-x  8 2000 2000 4096 Mar 30 10:39 .
drwxr-xr-x  3 bob  bob  4096 Mar 30 10:39 ..
drwxr-xr-x  2 2000 2000 4096 Mar 30 10:39 bleve-indexes
drwxr-xr-x  3 2000 2000 4096 Mar 30 10:39 client
drwxr-xr-x  2 2000 2000 4096 Mar 30 11:02 config
drwxr-xr-x 11 2000 2000 4096 Apr 17 17:47 data
drwxr-xr-x  2 2000 2000 4096 Apr  4 17:09 logs
drwxr-xr-x  5 2000 2000 4096 Apr 17 17:47 plugins

bob@newisa:~/mattermost/docker/volumes/app/mattermost/data$ ls -la
total 44
drwxr-xr-x 11 2000 2000 4096 Apr 17 17:47 .
drwxr-xr-x  8 2000 2000 4096 Mar 30 10:39 ..
drwxr-x---  3 2000 2000 4096 Apr  4 16:20 20240404
drwxr-x---  3 2000 2000 4096 Apr  5 12:35 20240405
drwxr-x---  3 2000 2000 4096 Apr  7 09:47 20240407
drwxr-x---  3 2000 2000 4096 Apr  8 10:26 20240408
drwxr-x---  3 2000 2000 4096 Apr  9 15:10 20240409
drwxr-x---  3 2000 2000 4096 Apr 16 15:47 20240416
drwxr-x---  3 2000 2000 4096 Apr 17 11:40 20240417
drwxr-x---  4 2000 2000 4096 Apr  4 11:43 teams
drwxr-x--- 15 2000 2000 4096 Apr 17 11:46 users

and given the permissions to see what’s in users I have to use sudo (as bob):

bob@newisa:~/mattermost/docker/volumes/app/mattermost/data$ sudo ls -l users
total 52
drwxr-x--- 2 2000 2000 4096 Apr 11 16:15 7zct6smsuinrxjejry9j8x77zr
drwxr-x--- 2 2000 2000 4096 Apr 11 16:15 apjcokqjsiye7bd8bconf1ixna
drwxr-x--- 2 2000 2000 4096 Mar 30 11:28 bn6gurcgzibwug1gpcxwp5y8uw
drwxr-x--- 2 2000 2000 4096 Apr  4 14:51 dodwmf88p3fsjn9bsntomzdpzh
drwxr-x--- 2 2000 2000 4096 Mar 30 11:02 ghno6ch7sidhmbknmjr1h744he
drwxr-x--- 2 2000 2000 4096 Apr  9 15:07 h8odzjfrpiygpjksghepmmca6y
drwxr-x--- 2 2000 2000 4096 Apr  5 17:13 jqaexyrpfbyezb3smraxqjdcmh
drwxr-x--- 2 2000 2000 4096 Apr  8 10:14 m779536izfr158iijziyw5hm1c
drwxr-x--- 2 2000 2000 4096 Mar 30 11:02 nmmdkys1jfn8fd84i34y4qumpy
drwxr-x--- 2 2000 2000 4096 Apr  5 12:34 rpjm3ykd97de5gkrqyfx5frtzc
drwxr-x--- 2 2000 2000 4096 Apr  5 13:52 tc4k8usf3ifybbpcbaf4nnnaha
drwxr-x--- 2 2000 2000 4096 Apr 17 11:46 w81gtbnf9inxmmj9y66egxbjgy
drwxr-x--- 2 2000 2000 4096 Apr  9 08:57 waid6boye3rameeqxaoe7f7uch

In the System Console → Environment → File Storage the first field has “Local File System” as its choice.

Expected behavior
Being able to upload PDFs, JPGs, Office files etc.

Observed behavior
The upload seems to progress with a progress bar but then I get a red “There was a problem uploading your files.” text under the box.

2

What does ~/mattermost/docker/volumes/app/mattermost/logs/mattermost.log say?

You may want/need to enable debug logging.

You could also log in to an interactive shell within the container to do some exploration with docker exec -ti mattermost-preview /bin/bash Try to touch a file, look at errors, etc.

3

Goodmorning and thank you so much for trying to help.
Started the client and attempted loading an image getting the usual error in the app.
Monitored the logfile you suggested with a
sudo tail -f ~/mattermost/docker/volumes/app/mattermost/logs/mattermost.log
but only see some startup errors (which are not as urgent right now):

{"timestamp":"2024-04-18 07:16:26.815 Z","level":"warn","msg":"Push notifications have been disabled. Update your license or go to System Console > Environment > Push Notification Server to use a different server","caller":"app/notification.go:33"}
{"timestamp":"2024-04-18 07:16:35.751 Z","level":"warn","msg":"Push notifications have been disabled. Update your license or go to System Console > Environment > Push Notification Server to use a different server","caller":"app/notification.go:33"}
{"timestamp":"2024-04-18 07:16:36.595 Z","level":"warn","msg":"Push notifications have been disabled. Update your license or go to System Console > Environment > Push Notification Server to use a different server","caller":"app/notification.go:33"}

so I incresed the log file level to DEBUG and restarted MM but as far as I can tell there’s nothing that gets logged when I try uploading the image:

{"timestamp":"2024-04-18 07:23:01.970 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","method":"GET","url":"/api/v4/users/me/teams/unread","request_id":"xkw3srjjajnizmojrtqo9y1gry","status_code":"200"}
{"timestamp":"2024-04-18 07:23:01.970 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","method":"GET","url":"/api/v4/users/me/teams/f159pjnrcffszfj1s41cbeto4a/channels","request_id":"jdnisk4nbigzmjhc8trkbyb5kr","status_code":"304"}
{"timestamp":"2024-04-18 07:23:01.971 Z","level":"debug","msg":"Received HTTP request","caller":"app/plugin_api.go:970","plugin_id":"playbooks","url":"/api/v0/runs?page=0&participant_id=m779536izfr158iijziyw5hm1c&per_page=0&team_id=f159pjnrcffszfj1s41cbeto4a","user_id":"m779536izfr158iijziyw5hm1c","request_id":"tbj891an7jf6dxfone6msduyme","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15","method":"GET","plugin_caller":"github.com/mattermost/mattermost-plugin-playbooks/server/api/logger.go:44"}
{"timestamp":"2024-04-18 07:23:01.974 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","method":"POST","url":"/api/v4/users/status/ids","request_id":"sgxarskqqjbgdnmgpap58885aa","status_code":"200"}
{"timestamp":"2024-04-18 07:23:01.976 Z","level":"debug","msg":"Handled HTTP request","caller":"app/plugin_api.go:970","plugin_id":"playbooks","url":"/api/v0/runs?page=0&participant_id=m779536izfr158iijziyw5hm1c&per_page=0&team_id=f159pjnrcffszfj1s41cbeto4a","user_id":"m779536izfr158iijziyw5hm1c","request_id":"tbj891an7jf6dxfone6msduyme","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15","method":"GET","time":"5","status":"200","plugin_caller":"github.com/mattermost/mattermost-plugin-playbooks/server/api/logger.go:57"}
{"timestamp":"2024-04-18 07:23:02.062 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","method":"GET","url":"/api/v4/channels/egsmb4ojf3nudykjniey784n7y/posts","request_id":"hgw5p8myzpd6ukzr81gid9p8pw","status_code":"200"}
{"timestamp":"2024-04-18 07:23:02.063 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","method":"GET","url":"/api/v4/config/client","request_id":"tdmmu1t1h3dnpdziofd5fnp5dy","status_code":"200"}
{"timestamp":"2024-04-18 07:23:02.068 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","method":"POST","url":"/api/v4/users/ids","request_id":"934whsueotgeinor3q9gbuxqxh","status_code":"200"}
{"timestamp":"2024-04-18 07:23:11.826 Z","level":"debug","msg":"Worker received a new candidate job.","caller":"resend_invitation_email/worker.go:67","worker":"resend_invitation_email"}
{"timestamp":"2024-04-18 07:23:18.866 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","method":"GET","url":"/api/v4/system/ping","request_id":"3ctpsjge1jb77b7wkowzuhhm5c","status_code":"200"}
{"timestamp":"2024-04-18 07:23:25.275 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","method":"POST","url":"/api/v4/users/status/ids","request_id":"fdnhpus8fbndxgnj5191nw7c4o","status_code":"200"}
{"timestamp":"2024-04-18 07:23:26.827 Z","level":"debug","msg":"Worker received a new candidate job.","caller":"resend_invitation_email/worker.go:67","worker":"resend_invitation_email"}
{"timestamp":"2024-04-18 07:23:28.134 Z","level":"debug","msg":"Websocket request","caller":"wsapi/websocket_handler.go:26","action":"user_update_active_status"}
{"timestamp":"2024-04-18 07:23:31.467 Z","level":"warn","msg":"Push notifications have been disabled. Update your license or go to System Console > Environment > Push Notification Server to use a different server","caller":"app/notification.go:33"}
{"timestamp":"2024-04-18 07:23:31.486 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","method":"POST","url":"/api/v4/channels/members/me/view","request_id":"zis6ixxxofnauq51qzd7mj9m4e","status_code":"200"}
{"timestamp":"2024-04-18 07:23:35.304 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","method":"POST","url":"/api/v4/channels/members/me/view","request_id":"nayxnfub8jngtfdmudr7or94fy","status_code":"200"}
{"timestamp":"2024-04-18 07:23:41.828 Z","level":"debug","msg":"Worker received a new candidate job.","caller":"resend_invitation_email/worker.go:67","worker":"resend_invitation_email"}
{"timestamp":"2024-04-18 07:23:48.915 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","method":"GET","url":"/api/v4/system/ping","request_id":"wq6kno5w87gppx7zbddx9bmdow","status_code":"200"}
{"timestamp":"2024-04-18 07:23:49.002 Z","level":"debug","msg":"Scheduling Job","caller":"migrations/scheduler.go:45","scheduler":"migrations"}
{"timestamp":"2024-04-18 07:23:49.002 Z","level":"debug","msg":"All migrations are complete.","caller":"migrations/scheduler.go:84","scheduler":"migrations"}
{"timestamp":"2024-04-18 07:23:49.003 Z","level":"debug","msg":"Next run time for scheduler","caller":"jobs/schedulers.go:147","scheduler_name":"migrations","next_runtime":"<nil>"}
{"timestamp":"2024-04-18 07:23:56.829 Z","level":"debug","msg":"Worker received a new candidate job.","caller":"resend_invitation_email/worker.go:67","worker":"resend_invitation_email"}
{"timestamp":"2024-04-18 07:24:01.385 Z","level":"debug","msg":"websocket.NextReader: closing websocket","caller":"platform/web_conn.go:849","user_id":"m779536izfr158iijziyw5hm1c","conn_id":"t6xt9tfq4bnn3d4eogbgnm4pza","error":"websocket: close 1006 (abnormal closure): unexpected EOF"}
{"timestamp":"2024-04-18 07:24:01.385 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","method":"GET","url":"/api/v4/websocket","request_id":"z6dqccxwciyx5byhkr7te7spgr"}
{"timestamp":"2024-04-18 07:24:05.455 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","method":"GET","url":"/api/v4/users/me/teams/f159pjnrcffszfj1s41cbeto4a/channels/members","request_id":"iqqen94kpbykby4s8k4m4k4a6w","status_code":"200"}
{"timestamp":"2024-04-18 07:24:05.491 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","method":"POST","url":"/api/v4/users/status/ids","request_id":"d31yg7rmsfy8tnqpuhjf5sg46r","status_code":"200"}
{"timestamp":"2024-04-18 07:24:05.492 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","method":"GET","url":"/api/v4/users/me/teams/unread","request_id":"f6nxike6cbfgjbgb4kch9pp9ew","status_code":"200"}
{"timestamp":"2024-04-18 07:24:05.493 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","method":"GET","url":"/api/v4/channels/pmix7sje4tre9b1xkru4tcdz8y/posts","request_id":"fkjex8t7b7re58o8ebyqpsfsxy","status_code":"200"}
{"timestamp":"2024-04-18 07:24:05.494 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","method":"GET","url":"/api/v4/users/me/teams/f159pjnrcffszfj1s41cbeto4a/channels","request_id":"zpc6u8rj5f8bzdh48zxbd8abya","status_code":"304"}
{"timestamp":"2024-04-18 07:24:05.550 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","method":"GET","url":"/api/v4/plugins/webapp","request_id":"3ygkmiwu5ffauyxkxkezboba3r","status_code":"200"}
{"timestamp":"2024-04-18 07:24:05.610 Z","level":"debug","msg":"Received HTTP request","caller":"app/plugin_api.go:970","plugin_id":"playbooks","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15","method":"GET","url":"/api/v0/runs?page=0&participant_id=m779536izfr158iijziyw5hm1c&per_page=0&team_id=f159pjnrcffszfj1s41cbeto4a","user_id":"m779536izfr158iijziyw5hm1c","request_id":"gicpweano3fk3ed6fueqn4arxh","plugin_caller":"github.com/mattermost/mattermost-plugin-playbooks/server/api/logger.go:44"}
{"timestamp":"2024-04-18 07:24:05.615 Z","level":"debug","msg":"Handled HTTP request","caller":"app/plugin_api.go:970","plugin_id":"playbooks","user_id":"m779536izfr158iijziyw5hm1c","request_id":"gicpweano3fk3ed6fueqn4arxh","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15","status":"200","time":"6","method":"GET","url":"/api/v0/runs?page=0&participant_id=m779536izfr158iijziyw5hm1c&per_page=0&team_id=f159pjnrcffszfj1s41cbeto4a","plugin_caller":"github.com/mattermost/mattermost-plugin-playbooks/server/api/logger.go:57"}
{"timestamp":"2024-04-18 07:24:05.617 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","method":"POST","url":"/api/v4/users/ids","request_id":"ze4b5j33y7rt8c3npmwed64deo","status_code":"200"}
{"timestamp":"2024-04-18 07:24:05.825 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","method":"GET","url":"/api/v4/channels/egsmb4ojf3nudykjniey784n7y/posts","request_id":"38opc8j7ypg89cowhpwox78awo","status_code":"200"}

I also tried exploring inside the container but not able to see anything of significance :frowning:

I see other people had similar problems and solved them with a fresh reinstall but as I have a few active users and posts on the server I would really want to avoid having them to restart it all

Other ideas? Again thanks a lot.
Robert

4

What about opening the interactive shell and trying to touch a file? See what outputs you get at the console or in the system logs within the container.

Do you have SELinux or AppArmor or anything like that running on your host OS? What do the system logs for it say?

This definitely sounds an awful lot like the user that owns the Mattermost process in the container is not able to write to the volume mount.

Also… it sounds like this was working and then stopped at some point. What happened at that time?

1 Like
5

Thank you John and sorry for the late reply as I’ve been under pressure for a project deadline.

Are you suggesting I open a shell in the container (docker compose exec -it SERVICENAME /bin/bash ?

If so in which directories would it be worth testing a file creation/deletion/modification?

No I do not have any security extensions. A vanilla Debian 11.

Take care. Robert

6

Mattermost writes files to /opt/mattermost/data I’d try to touch a file there. I’m sure there’s some sort of Docker troubleshooting tool that would show volume mounts and permissions. That’s my best guess… there is something beyond simple file permissions and the Docker volume mount. I’m just not smart enough to know much more!

7

Do you checked the nginx configuration of docker/docker-compose.nginx.yml at main · mattermost/docker (github.com)
with your own external nginx configurtion? It might be that file writing is not even reaching the server behind the nginx, thats why also no logs are showing.
You could check your browser developer tools log if there is something telling you about errors or issues (F12 in the most browsers or when you use the app: docker/docker-compose.nginx.yml at main · mattermost/docker (github.com)

1 Like