I’ve struggled my way through all of the import steps lol but now i seem to be hitting a brick wall of sorts. when attempting to import the ./mattermost-bulk-import.zip it sits for quite a while and then i get the following:
Error: failed to upload data: AppErrorFromJSON: model.utils.decode_json.app_error, body:
There is nothing more than that shown for the error. I am doing this all from a windows machine with Mattermost installled on a Turnkey Linux VM on an unraid server - Not sure any of this matters just wanted to give my setup in case it was useful.
The “mattermost-bulk-import.zip” file is comprised of: mattermost_import.jsonl then a subfolder called “data” within data another folder “bulk-export-attachments” and within that folder are all the attachments.
I have updated the maximum file size within my mattermost server itself and the file is 4.6GB I had not updated the NGINX maximum file size as I am not using NGINX integrated with Mattermost but rathern an NGINX Docker - I do not see any settings on the docker whatsoever for maximum file size or anything like that so I’m thinking that NGINX isnt the issue at hand here.
Sorry for the long winded message trying to outline every possible thing so the solution is easier for someone who knows more than I
The sed command only works if you have the nginx config file available. In your case, you said you’re using an nginx container and the client_max_body_size parameter needs to be changed in the container’s nginx config.
What container are you using and how do you start it? Maybe there’s an environment variable or the possibility to add configuration options from “the outside” so that the config gets applied properly.
If you’re not using the Mattermost-integrated nginx container you most likely also had to create a configuration file for the Mattermost VHOST there - the server section in this config file would also be a good location to override the default client_max_body_size of the container’s nginx setup.
Thank you for the response. Maybe a little more explanation on my setup is warranted here. I am using Unraid and have mattermost installed in a linux VM using TurnkeyLinux Mattermost ISO. From the notes at Turnkey Linux: “Includes Nginx (webserver); pre-configured to proxy Mattermost.” I guess I am unsure how to access or edit this Nginx webserver that is apparently preconfigured.
What I am doing for NGINX: I have a docker installed on Unraid NginxProxyManager and i simply used that to reverse proxy the IP of the VM rather than trying to figure out how to do it through the included Nginx within the mattermost installation.
My presumption is even though I am not using the preconfigured Nginx Webserver that somehow this is what is limiting my file size.
After using Turnkeylinux “webmin” panel and going to file manager i was able to locate in a folder called “etc” an nginx folder. within there i have several files and subfolders. Hoping maybe i can simply edit one of those files.
To further this on the webmin panel when i go to this nginx folder i can “open the current path in terminal” i attempted to do that and simply add the sed command from the other post but no luck
within the nginx folder there is a conf.d folder – however there is no “mattermost.conf” file within that folder.
After looking around online more and browsing the file structure I do have a mattermost.conf file in /etc/nginx/sites-available/mattermost — Based on this i changed my “sed” command to read as follows:
sed -i -e 's/50M/6G/' /etc/nginx/sites-available/mattermost
Then i ran
systemctl reload nginx
When i enter both of these codes in i dont get errors or anything and the code appears to work. I manually checked the mattermost.conf file and it shows “client_max_body_size 6G” so it seems as though my commands didd indeed work. However the file import still fails with the same error.
just thinking out loud here… Wondering if i can manually import the file via the webmin interface. I tested and I am able to upload it – but where exactly to upload it is what i am unsure on
Hi again and sorry for the late reply, I was busy the last days.
Yes, this file sounds correct, although it’s not necessarily the right file and there might be multiple occurences of this configuration option.
Can you please login via SSH to your system and run the following command as root (or via sudo)?
I’ve also added an example from one of my systems, so that you can see what it looks like.
On my system, as you can see, the client_max_body_size parameter is specified three times, so you would have to change it in all places, if it’s there more than once on your system:
root@ag-ubuntu0101:~# eval /proc/$(pidof -s nginx)/exe -T | grep client_max_body
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
client_max_body_size 50M;
client_max_body_size 50M;
client_max_body_size 50M;
You can upload the file directly to your import folder. It’s directly below the data folder, which is /opt/mattermost/data by default, so you should see a folder called /opt/mattermost/data/import. The mmctl upload process uploads the file to this directory and it also suffixes it with a job ID, but I think this is not really necessary, so a manual upload should work too.
in regards to directly uploading the folder: upon entering /opt/mattermost/data there is no import folder. I created an import folder and directly uploaded the file to said folder.
After doing that i ran mmctl import list available and to my surprise the file showed up as mattermost-bulk-import.zip
I then ran mmctl import process mattermost-bulk-import.zip and te results were Import porcess job sucessfully created, ID: ggz3gqf1atbrbb7ymgh917r3te
Then i ran mmctl import job show ggz3gqf1atbrbb7ymgh917r3te --json and the results were:
[
{
"id": "ggz3gqf1atbrbb7ymgh917r3te",
"type": "import_process",
"priority": 0,
"create_at": 1663069649437,
"start_at": 1663069663702,
"last_activity_at": 1663069663744,
"status": "error",
"progress": -1,
"data": {
"error": "We could not count the users. - ImportProcessWorker: Unable to process import: failed to open file., zip: not a valid zip file",
"import_file": "mattermost-bulk-import.zip"
}
}
]
So now its saying the file is not a valid zip file???
OK, thanks - this confirms that the client_max_body_size has been properly set to 6G, but I think you will also have to increase the maximum file size in the System console, but I’m not 100% sure about that, but it doesn’t hurt to increase this value for testing purposes:
that did work but i still cannot do the mmctl import process…
When i run mmctl import job show i get back the following error:
We could not count the users. - ImportProcessWorker: Unable to process import: failed to open file., zip: not a valid zip file
This is on the manually uploaded file through the UI. I still cannot upload via mmctl i get the error that it is too large.
I hope i can get this working and really appreciate your help! As soon as i can get this done i plan to move entirely from slack but cant until i get historical info in there.
Can you verify that the .zip in opt/mattermost/data/import/ is the same size as the one you’ve uploaded? If the file was somehow truncated during the upload, it will be invalid. The next version of mmctl will contain an import validator, which should catch those problems earlier.
When i run this command i get could not find: /opt/mattermost/data/import/mattermost-bulk-import.z01
If i go and MMCTL and hit import list available i do see: mattermost-bulk-import.zip
If I attempt to run mmctl import job show ID --json i get unable to process import: failed to open file., zip: not a valid zip file"
And this is still on the manually uploaded (through webmin) file - If i try and upload via mmctl import it errors out with the same error listed above that it is too large.
I did double check the file size and it is perfect. I dont think the next version of mmctl will solve this issue as I’m not able to use the import command to import the file for some reason and i have to manually upload the file via the webmin console of my turnkey linux mattermost VM
That’s interesting - it thinks you have a splitfile.
Something is definitely wrong with this file, but maybe it’s a problem with the wildcard, so one more try, without the wildcard, please:
zip -T /opt/mattermost/data/import/mattermost-bulk-import.zip
If this also asks you for a .z01 file, then we might have a problem here.
Can you try to unzip the file on the Mattermost server then (just for testing purposes)?
cd /opt/mattermost/data/import
unzip mattemorst-bulk-import.zip
