Unable to upload files

Troubleshooting
1

Summary
I’m not able to upload any images.

Steps to reproduce
If I try to upload a profile picture I get the error message “Couldn’t upload profile image.”

The same applies if I try to upload an image in a post (without the error message).

Expected behavior
No error message :wink:

Observed behavior

CleanShot 2023-02-18 at 22.46.34@2x
CleanShot 2023-02-18 at 22.46.34@2x1182×576 29.1 KB

Here is what is written in the log:
{“timestamp”:“2023-02-18 21:45:21.934 Z”,“level”:“error”,“msg”:“Couldn’t upload profile image.”,“caller”:“web/context.go:117”,“path”:“/api/v4/users/r9jbasyaciyefbfp6shf5mc8cy/image”,“request_id”:“jdqsutbyn7rx3dw6huar11mctr”,“ip_addr”:“127.0.0.1”,“user_id”:“r9jbasyaciyefbfp6shf5mc8cy”,“method”:“POST”,“err_where”:“SetProfileImage”,“http_code”:500,“error”:“SetProfileImage: Couldn’t upload profile image., WriteFile: Unable to write the file., unable to create the directory /var/opt/mattermost/data/users/r9jbasyaciyefbfp6shf5mc8cy for the file /var/opt/mattermost/data/users/r9jbasyaciyefbfp6shf5mc8cy/profile.png: mkdir /var/opt/mattermost/data: permission denied”}
{“timestamp”:“2023-02-18 21:45:21.989 Z”,“level”:“error”,“msg”:“Unable to write the file.”,“caller”:“web/context.go:117”,“path”:“/api/v4/users/r9jbasyaciyefbfp6shf5mc8cy/image”,“request_id”:“4f7mundbr3dz8xwsrsuf79d6ww”,“ip_addr”:“127.0.0.1”,“user_id”:“r9jbasyaciyefbfp6shf5mc8cy”,“method”:“GET”,“err_where”:“WriteFile”,“http_code”:500,“error”:“WriteFile: Unable to write the file., unable to create the directory /var/opt/mattermost/data/users/r9jbasyaciyefbfp6shf5mc8cy for the file /var/opt/mattermost/data/users/r9jbasyaciyefbfp6shf5mc8cy/profile.png: mkdir /var/opt/mattermost/data: permission denied”}

2

having the same problem

3

I deleted the whole server instance and did a complete re-install. Now it is working :wink:

4

Hi @scrumphony and welcome to the Mattermost forums!

Sorry for my late reply, but the error message indicates that the file permissions were wrong, so maybe upon your first try, you forgot to run the necessary chown command on the /var/opt/mattermost directory.

@lucyclhill can you check that on your system, since you seem to have the same problem? If your Mattermost application is being run with user mattermost (f.ex.), then the directory and all the files below it also need to be owned by this user. The default installation suggests to use /opt/mattermost as the base directory as well as mattermost for the system account, so if you changed it, please do also change it in the following command:

chown -R mattermost: /opt/mattermost

This should fix all issues related to wrong filesystem permissions.

5

Hi @agriesser

Thank you for the reply. This was the strange thing: the file permissions were correct. I even did a chmod -r 777 on the directory, but that didn’t help either.

Now, that I installed everything from scratch everything is running smoothly and my community enjoys using Mattermost :rocket:

6

Very interesting - sorry for not having replied earlier, would have been an interesting debugcase. Anyways, I’m marking this as resolved now.

7

hi there - I’m not a tech / IT person so I’m afraid I don’t know what this request means / what I’m meant to do… can you give itemised instructions on how I do this?

@lucyclhill can you check that on your system, since you seem to have the same problem? If your Mattermost application is being run with user mattermost (f.ex.), then the directory and all the files below it also need to be owned by this user. The default installation suggests to use /opt/mattermost as the base directory as well as mattermost for the system account, so if you changed it, please do also change it in the following command:

8

I did send the command you need to run right after the text part you quoted, but it’s hard for me to give you detailed itemised instructions without knowing anything about your deployment. One of the good things with Mattermost is, that you can install it in many different ways and it integrates into many different applications and this is also (from a support point of view) a very challenging situation, since the same fix needs to be carried out differently depending on the context.

What deployment method did you use? Docker? Binary? Omnibus? Gitlab-integrated? etc.

9

Hi, I’m having the same problem. I installed mattermost-omnibus which set me as File Storage System: Local File System and as Local Storage Directory /var/opt/mattermost/data/ with the command “chown -R mattermost: /opt/mattermost” I get the response on all files “chown: changing ownership of ‘/opt/mattermost/i18n/en_AU.json’: Operation not permitted” Do you have any suggestions?

10

Hi @AndreaGalli and welcome to the Mattermost forums!

Did you run the chown command as root or using sudo?

11

Also having this issue. The mattermost directory and all subdirectories show mattermost as the owner (and re-ran chown -R mattermost just in case). Permissions are set as:
drwxr-x— 3 mattermost mattermost 4,0K Jun 26 07:38 20230626

Tried disabling and re-enabling file sharing in the console, but still not happening – uploads just hang on “Processing…”

12

Hi Carolyna and welcome to the Mattermost forums!

What Mattermost server version are you on currently? There has been a bug that has been fixed with the 7.8 release of the application that was causing file uploads to be stuck in “Processing” state irregularly.
Another reason for such behaviour could be an Antivirus plugin - did you install one in your Mattermost installation?

13

I am experiencing this problem on server version 7.10.4.

  • Install method: Omnibus
  • Data Directory Permissions: drwxrwxrwx 5 mattermost mattermost 4096 Jul 16 13:22 data
  • No antivirus

I did a test where I modified the data directory back to the default path (/var/opt/mattermost/data) and the attachments work as expected. On this directory, the permissions are:
drwxr-xr-x 5 mattermost mattermost 4096 Jul 19 07:25 data

15

I just experienced the same error message…

The avatar images are stored in /var/opt/mattermost/data/users/, and application related data is stored in /opt/mattermost/ (mind the small differences in the path).

Thus, in order to solve this particular issue, please execute the following in the terminal:

sudo chown -R mattermost:mattermost /var/opt/mattermost/

Checking the permissions with ls -la /var/opt/mattermost/ should look like this:

root@app:/var/opt/mattermost/data# ls -la /var/opt/mattermost/
total 16
drwx------ 4 mattermost mattermost 4096 Aug 27 10:24 .
drwxr-xr-x 3 root       root       4096 Aug 27 10:24 ..
drwx------ 2 mattermost mattermost 4096 Aug 27 10:24 backups
drwxrwxr-x 3 mattermost mattermost 4096 Aug 27 10:48 data

PS: Re-installing the whole server instance is basically never a good solution due to the complete loss of data… :joy:

16

I am having this issue as well, i installed it with Omnibus, and i have set the storage as S3 (and i have tested the connection and it is valid).

root@mattermost:/var/opt/mattermost/data/users/xe5uzemdcfgnjbi1e8ejimk11o# tail -f /var/log/mattermost/mattermost.log
{“timestamp”:“2023-09-20 13:18:42.198 -07:00”,“level”:“error”,“msg”:“Unable to write the file.”,“caller”:“web/context.go:117”,“path”:“/api/v4/users/d1pe59ntafyuzcjbesr7ucpoie/image”,“request_id”:“ncbwudrwtbnzzy55jj96iryfwy”,“ip_addr”:“127.0.0.1”,“user_id”:“n79i9ceknpddz8xh6fh1wf5cnr”,“method”:“GET”,“err_where”:“WriteFile”,“http_code”:500,“error”:“WriteFile: Unable to write the file., unable write the data in the file users/d1pe59ntafyuzcjbesr7ucpoie/profile.png: Access Denied”}
{“timestamp”:“2023-09-20 13:18:42.289 -07:00”,“level”:“error”,“msg”:“Unable to write the file.”,“caller”:“web/context.go:117”,“path”:“/api/v4/users/hprgfdwsbpdbirum36fjrq1wiy/image”,“request_id”:“ipphy7rnr38odd5kz54agziqao”,“ip_addr”:“127.0.0.1”,“user_id”:“n79i9ceknpddz8xh6fh1wf5cnr”,“method”:“GET”,“err_where”:“WriteFile”,“http_code”:500,“error”:“WriteFile: Unable to write the file., unable write the data in the file users/hprgfdwsbpdbirum36fjrq1wiy/profile.png: Access Denied”}
{“timestamp”:“2023-09-20 13:18:42.299 -07:00”,“level”:“error”,“msg”:“Unable to write the file.”,“caller”:“web/context.go:117”,“path”:“/api/v4/users/sz6rj9uag38bmyfk9zi3kza71a/image”,“request_id”:“tbkyctn85pfhdn8w467cja3jaw”,“ip_addr”:“127.0.0.1”,“user_id”:“n79i9ceknpddz8xh6fh1wf5cnr”,“method”:“GET”,“err_where”:“WriteFile”,“http_code”:500,“error”:“WriteFile: Unable to write the file., unable write the data in the file users/sz6rj9uag38bmyfk9zi3kza71a/profile.png: Access Denied”}
{“timestamp”:“2023-09-20 13:18:44.444 -07:00”,“level”:“error”,“msg”:“Unable to write the file.”,“caller”:“web/context.go:117”,“path”:“/api/v4/users/xe5uzemdcfgnjbi1e8ejimk11o/image”,“request_id”:“8rym35q9i3rfmfdz8dfg74t1kc”,“ip_addr”:“127.0.0.1”,“user_id”:“n79i9ceknpddz8xh6fh1wf5cnr”,“method”:“GET”,“err_where”:“WriteFile”,“http_code”:500,“error”:“WriteFile: Unable to write the file., unable write the data in the file users/xe5uzemdcfgnjbi1e8ejimk11o/profile.png: Access Denied”}
{“timestamp”:“2023-09-20 13:18:44.583 -07:00”,“level”:“error”,“msg”:“Unable to write the file.”,“caller”:“web/context.go:117”,“path”:“/api/v4/users/d1pe59ntafyuzcjbesr7ucpoie/image”,“request_id”:“dsxfrpewctytfrhr369zk5ti7e”,“ip_addr”:“127.0.0.1”,“user_id”:“n79i9ceknpddz8xh6fh1wf5cnr”,“method”:“GET”,“err_where”:“WriteFile”,“http_code”:500,“error”:“WriteFile: Unable to write the file., unable write the data in the file users/d1pe59ntafyuzcjbesr7ucpoie/profile.png: Access Denied”}
{“timestamp”:“2023-09-20 13:18:48.630 -07:00”,“level”:“error”,“msg”:“Unable to write the file.”,“caller”:“web/context.go:117”,“path”:“/api/v4/users/r7rdf5kyeirmxqcuf4nfm7pauc/image”,“request_id”:“xnd3jqsqx3fj98rceger5ggwtw”,“ip_addr”:“127.0.0.1”,“user_id”:“n79i9ceknpddz8xh6fh1wf5cnr”,“method”:“GET”,“err_where”:“WriteFile”,“http_code”:500,“error”:“WriteFile: Unable to write the file., unable write the data in the file users/r7rdf5kyeirmxqcuf4nfm7pauc/profile.png: Access Denied”}
{“timestamp”:“2023-09-20 13:18:48.831 -07:00”,“level”:“error”,“msg”:“Unable to write the file.”,“caller”:“web/context.go:117”,“path”:“/api/v4/users/sz6rj9uag38bmyfk9zi3kza71a/image”,“request_id”:“6caxndksk3gzibo4h96fzjx53a”,“ip_addr”:“127.0.0.1”,“user_id”:“n79i9ceknpddz8xh6fh1wf5cnr”,“method”:“GET”,“err_where”:“WriteFile”,“http_code”:500,“error”:“WriteFile: Unable to write the file., unable write the data in the file users/sz6rj9uag38bmyfk9zi3kza71a/profile.png: Access Denied”}
{“timestamp”:“2023-09-20 13:18:48.836 -07:00”,“level”:“error”,“msg”:“Unable to write the file.”,“caller”:“web/context.go:117”,“path”:“/api/v4/users/d1pe59ntafyuzcjbesr7ucpoie/image”,“request_id”:“qc9rsstp63nsikoieejbu97c8w”,“ip_addr”:“127.0.0.1”,“user_id”:“n79i9ceknpddz8xh6fh1wf5cnr”,“method”:“GET”,“err_where”:“WriteFile”,“http_code”:500,“error”:“WriteFile: Unable to write the file., unable write the data in the file users/d1pe59ntafyuzcjbesr7ucpoie/profile.png: Access Denied”}
{“timestamp”:“2023-09-20 13:18:48.916 -07:00”,“level”:“error”,“msg”:“Unable to write the file.”,“caller”:“web/context.go:117”,“path”:“/api/v4/users/hprgfdwsbpdbirum36fjrq1wiy/image”,“request_id”:“df1yxw7uzpyzjxj1tiwk7czqte”,“ip_addr”:“127.0.0.1”,“user_id”:“n79i9ceknpddz8xh6fh1wf5cnr”,“method”:“GET”,“err_where”:“WriteFile”,“http_code”:500,“error”:“WriteFile: Unable to write the file., unable write the data in the file users/hprgfdwsbpdbirum36fjrq1wiy/profile.png: Access Denied”}
{“timestamp”:“2023-09-20 13:18:59.798 -07:00”,“level”:“error”,“msg”:“Couldn’t upload profile image.”,“caller”:“web/context.go:117”,“path”:“/api/v4/users/n79i9ceknpddz8xh6fh1wf5cnr/image”,“request_id”:“pcbcd1tbsj8kbbn56hcuqta67h”,“ip_addr”:“127.0.0.1”,“user_id”:“n79i9ceknpddz8xh6fh1wf5cnr”,“method”:“POST”,“err_where”:“SetProfileImage”,“http_code”:500,“error”:“SetProfileImage: Couldn’t upload profile image., WriteFile: Unable to write the file., unable write the data in the file users/n79i9ceknpddz8xh6fh1wf5cnr/profile.png: Access Denied”}
{“timestamp”:“2023-09-20 13:19:12.206 -07:00”,“level”:“error”,“msg”:“Couldn’t upload profile image.”,“caller”:“web/context.go:117”,“path”:“/api/v4/users/n79i9ceknpddz8xh6fh1wf5cnr/image”,“request_id”:“j77w3tk4f3nxme43ebcymjprda”,“ip_addr”:“127.0.0.1”,“user_id”:“n79i9ceknpddz8xh6fh1wf5cnr”,“method”:“POST”,“err_where”:“SetProfileImage”,“http_code”:500,“error”:“SetProfileImage: Couldn’t upload profile image., WriteFile: Unable to write the file., unable write the data in the file users/n79i9ceknpddz8xh6fh1wf5cnr/profile.png: Access Denied”}
root@mattermost:/var/opt/mattermost/data/users/xe5uzemdcfgnjbi1e8ejimk11o# ls -la /var/opt/mattermost/
total 16
drwxrwxr-x 4 mattermost mattermost 4096 Sep 9 05:19 .
drwxr-xr-x 3 root root 4096 Sep 9 05:04 …
drwxrwxr-x 2 mattermost mattermost 4096 Sep 18 11:55 backups
drwxrwxr-x 9 mattermost mattermost 4096 Sep 17 21:11 data
root@mattermost:/var/opt/mattermost/data/users/xe5uzemdcfgnjbi1e8ejimk11o#

17

This seems to be a Mattermost issue and not permission issue, i had the same issue using Omniinstall, so i set up a new server and are using the tarball this time. Both is having the same issue.
It seems to work fine for 2-3 days and then it broke.
I am running Mattermost of Ubuntu 22, hosted on Hetzner
ImageProxy is disabled, and i have set S3 as the storage places, i have also tried with local storage. Not difference.

root@mattermost:/opt/mattermost/data# sudo su - mattermost
su: warning: cannot change directory to /home/mattermost: No such file or directory
$ ls
20230921  20230922  20230928  brand  users
$ pwd
/opt/mattermost/data
$ ls -la
total 28
drwxrwxrwx  7 mattermost mattermost 4096 Sep 28 19:05 .
drwxrwxr-x 12 mattermost mattermost 4096 Sep 20 22:11 ..
drwxrwxrwx  3 mattermost mattermost 4096 Sep 21 01:01 20230921
drwxrwxrwx  3 mattermost mattermost 4096 Sep 22 00:59 20230922
drwxrwxrwx  2 root       root       4096 Sep 28 19:05 20230928
drwxrwxrwx  2 mattermost mattermost 4096 Sep 20 22:54 brand
drwxrwxrwx 11 mattermost mattermost 4096 Sep 22 18:01 users
$ cd u
-sh: 4: cd: can't cd to u
$ cd users
$ ls
4wztbwmea3rcde3j1dp4djb19c  ctzb1mtq3bggjkrxwetwubxknc  hprgfdwsbpdbirum36fjrq1wiy  r7rdf5kyeirmxqcuf4nfm7pauc  xe5uzemdcfgnjbi1e8ejimk11o
5umy9eogr7ntffmezcqphfco7o  d1pe59ntafyuzcjbesr7ucpoie  n79i9ceknpddz8xh6fh1wf5cnr  sz6rj9uag38bmyfk9zi3kza71a
$ cd 4wztbwmea3rcde3j1dp4djb19c
$ ls
profile.png
$ pwd
/opt/mattermost/data/users/4wztbwmea3rcde3j1dp4djb19c
$ less profile.png
"profile.png" may be a binary file.  See it anyway?
$ touch testfile
$ ls
profile.png  testfile
$ pwd
/opt/mattermost/data/users/4wztbwmea3rcde3j1dp4djb19c

The Application is running as Mattermost:

root@mattermost:/opt/mattermost/data# ps aux | grep mattermost
matterm+  122078  0.1  3.4 1870680 134792 ?      Ssl  Sep28   8:18 /opt/mattermost/bin/mattermost
matterm+  122088  0.0  0.5 729396 20236 ?        Sl   Sep28   0:31 plugins/com.mattermost.nps/server/dist/plugin-linux-amd64
matterm+  122096  0.7  0.8 736436 34740 ?        Sl   Sep28  39:50 plugins/com.mattermost.calls/server/dist/plugin-linux-amd64
root      184144  0.0  0.0   6608  2236 pts/0    S+   21:21   0:00 grep --color=auto mattermost

root@mattermost:/opt/mattermost/data# systemctl status mattermost
● mattermost.service - Mattermost
     Loaded: loaded (/lib/systemd/system/mattermost.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-09-28 01:23:52 UTC; 3 days ago
   Main PID: 122078 (mattermost)
      Tasks: 29 (limit: 4523)
     Memory: 254.7M
        CPU: 48min 41.067s
     CGroup: /system.slice/mattermost.service
             ├─122078 /opt/mattermost/bin/mattermost
             ├─122088 plugins/com.mattermost.nps/server/dist/plugin-linux-amd64
             └─122096 plugins/com.mattermost.calls/server/dist/plugin-linux-amd64

Oct 01 21:23:06 mattermost mattermost[122078]: {"timestamp":"2023-10-01 21:23:06.514 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","m>
Oct 01 21:23:06 mattermost mattermost[122078]: {"timestamp":"2023-10-01 21:23:06.519 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","m>
Oct 01 21:23:06 mattermost mattermost[122078]: {"timestamp":"2023-10-01 21:23:06.524 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","m>
Oct 01 21:23:06 mattermost mattermost[122078]: {"timestamp":"2023-10-01 21:23:06.554 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","m>
Oct 01 21:23:06 mattermost mattermost[122078]: {"timestamp":"2023-10-01 21:23:06.554 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","m>
Oct 01 21:23:10 mattermost mattermost[122078]: {"timestamp":"2023-10-01 21:23:10.675 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:163","m>
Oct 01 21:23:22 mattermost mattermost[122078]: {"timestamp":"2023-10-01 21:23:22.304 Z","level":"debug","msg":"Websocket request","caller":"wsapi/websocket_handler.go:>
Oct 01 21:23:22 mattermost mattermost[122078]: {"timestamp":"2023-10-01 21:23:22.307 Z","level":"debug","msg":"Websocket request","caller":"wsapi/websocket_handler.go:>
Oct 01 21:23:38 mattermost mattermost[122078]: {"timestamp":"2023-10-01 21:23:38.209 Z","level":"debug","msg":"Websocket request","caller":"wsapi/websocket_handler.go:>
Oct 01 21:23:38 mattermost mattermost[122078]: {"timestamp":"2023-10-01 21:23:38.209 Z","level":"debug","msg":"Websocket request","caller":"wsapi/websocket_handler.go:>

It is not a disk space issue:

root@mattermost:/opt/mattermost/data# df -h /opt/mattermost/data
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        75G  4.1G   68G   6% /

I also tried if it was using the /var/mattermost/data instead of /opt/mattermost/data, same issue:

root@mattermost:/opt/mattermost/data# mkdir -p /var/opt/mattermost/data
root@mattermost:/var/opt/mattermost/data# sudo chown mattermost:mattermost -R /var/opt/mattermost/data
root@mattermost:/var/opt/mattermost/data# sudo chmod 700 -R /var/opt/mattermost/data
root@mattermost:/var/opt/mattermost# ls -la
total 12
drwxr-xr-x 3 root       root       4096 Oct  1 21:47 .
drwxr-xr-x 3 root       root       4096 Oct  1 21:47 ..
drwx------ 2 mattermost mattermost 4096 Oct  1 21:47 data

The working directory is correct:

root@mattermost:/opt/mattermost/config# ls -l /proc/655/cwd
lrwxrwxrwx 1 mattermost mattermost 0 Oct  1 22:02 /proc/655/cwd -> /opt/mattermost
root@mattermost:/opt/mattermost/config# pwdx 655
655: /opt/mattermost

File settings:

    "FileSettings": {
        "EnableFileAttachments": true,
        "EnableMobileUpload": true,
        "EnableMobileDownload": true,
        "MaxFileSize": 104857600,
        "MaxImageResolution": 33177600,
        "MaxImageDecoderConcurrency": -1,
        "DriverName": "amazons3",
        "Directory": "./data/",
        "EnablePublicLink": false,
        "ExtractContent": true,
        "ArchiveRecursion": false,
        "PublicLinkSalt": "xxxxxxxxxxxxxxxxx",
        "InitialFont": "nunito-bold.ttf",
        "AmazonS3AccessKeyId": "xxxxxxxxxxxxxx",
        "AmazonS3SecretAccessKey": "xxxxxxxxxxxxxxxxxx",
        "AmazonS3Bucket": "xxxxxxxxxxxxxxx",
        "AmazonS3PathPrefix": "",
        "AmazonS3Region": "us-east-1",
        "AmazonS3Endpoint": "s3.amazonaws.com",
        "AmazonS3SSL": false,
        "AmazonS3SignV2": false,
        "AmazonS3SSE": false,
        "AmazonS3Trace": false,
        "AmazonS3RequestTimeoutMilliseconds": 30000,
        "DedicatedExportStore": false,
        "ExportDriverName": "local",
        "ExportDirectory": "./data/",
        "ExportAmazonS3AccessKeyId": "",
        "ExportAmazonS3SecretAccessKey": "",
        "ExportAmazonS3Bucket": "",
        "ExportAmazonS3PathPrefix": "",
        "ExportAmazonS3Region": "",
        "ExportAmazonS3Endpoint": "s3.amazonaws.com",
        "ExportAmazonS3SSL": true,
        "ExportAmazonS3SignV2": false,
        "ExportAmazonS3SSE": false,
        "ExportAmazonS3Trace": false,
        "ExportAmazonS3RequestTimeoutMilliseconds": 30000,
        "ExportAmazonS3PresignExpiresSeconds": 21600
    },

Update: I just changed the file upload to Local Storage and restarted the server, it seems like the upload is working now. So the issue must be with the S3 integration.