I’m not sure I understand exactly what you are saying here. No where along the lines have i used -s that i can tell??
I explained above but maybe i did something incorrect here with the zip file??? Here is what my zip file looks like:
When you browse into the zip file, you can see the contents on Windows but it does not necessarily guarantee that you can also extract it properly if parts are missing.
For whatever reason, your zip file has been split into multiple pieces and/or is corrupt.
So please explain how you created the mattermost-bulk-import.zip file in the first place (share the exact command, please). How did you get the zip file to your windows host? Did you maybe download it from the Mattermost server and then re-uploaded it via webmin to it using your Windows host? Did you, by any chance, use FTP for the transfer and could it be that you used the wrong transfer mode (ASCII vs. BINARY)?
Your zip file is exactly 4.5GB in size and probably during the creation, additional files have been created which have not been transferred.
How big is the data directory on the source (the server where you created the zip)?
Did I get it right that the slack export, the outputs of the mmetl command and the zipping of the mattermost-bulk-import.zip file did not happen directly on the Mattermost server but on a different linux machine? If so, can you please share the output of the following commands run on this machine?
cd /path/to/the/directory/containing/mattermost-bulk-import-and-mattermost-import-jsonl
du -sh data
ls -lh
I dont recall the exact steps and commands taken so what i will do is go ahead go through everything and recreate this file and document my steps along the way and update you once i’ve done that.
Thanks so far you’ve been a huge help
1 Like
The last three commands can be run without having to recreate everything and it will help us understand what your zip file looks like and if there are any *.z0 files, but maybe it’s good to start over new from the beginning and to document everything and let us know what’s wrong then at what point.
OK sorry for the long delay i wanted to go through each and every step line by line from the beginning and document my steps. Unfortunately I’m left with virtually the same exact problem.
Step1: Went to Slack and created an export for the entire date range of my slack history. February 10th 2020 to September 19th 2022
Export Downloaded as a zip file with 869 KB
Step2: Download all Slack files and attachments by running the following commands:
slack-advanced-exporter --input-archive "C:\Users\Adam\Desktop\mattermost\myfilename.zip" --output-archive export-with-emails.zip fetch-emails --api-token mytoken here
slack-advanced-exporter --input-archive export-with-emails.zip --output-archive export-with-emails-and-attachments.zip fetch-attachments
Upon Completion of these commands i am left with two files.
File 1: export-with-emails.zip 901 KB
File 2: export-with-emails-and-attachments.zip 4,646,025 KB
**NOTE** When creating the file some come up file share post has missing properties on its file object. Not many just a few
Step3: mmetl transform slack --team slackimport --file export-with-emails-and-attachments.zip --output mattermost_import.jsonl
The command completes with "Transformation succeeded!!" - **NOTE** Scrolling through i do notice several "Error retrieving file with ID xxxxx"
**NOTE** the mattermost_import.jsonl file is 913 KB
Step4: create a new folder "data" and move "bulk-export-attachments" into this folder. bulk-export-attachments is 4.80 GB or 5,157,861,533 Bytes
Step5: create zip file named "mattermost-bulk-import" with the mattermost_import.jsonl and the data folder created in the previous step. Did this by selecting the .json1 and the data folder and clicking "send to compressed folder"
**NOTE** mattermost-bulk-import.zip is 4.42 GB or 4,645,683 KB
Step6: ran command mmctl import upload ./mattermost-bulk-import.zip
command initiated at 7:57 AM upload failed at 8:59 AM
C:\Users\Adam\Desktop\mattermost>mmctl import upload ./mattermost-bulk-import.zip
Error: failed to upload data: AppErrorFromJSON: model.utils.decode_json.app_error, body: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>cloudflare</center>
</body>
</html>
Step7: upon failing of mattermost file upload I manually moved the mattermost-bulk-import.zip file over to the VM - /opt/mattermost/data/import
Step8: ran command mmctl import list available --- results mattermost-bulk-import.zip
Step9: ran command mmctl import process mattermost-bulk-import.zip
import process job successfully created, ID: sd1adtgsiffc3yt7shder3qeaw
Step10: C:\Users\Adam\Desktop\mattermost>mmctl import job show sd1adtgsiffc3yt7shder3qeaw --json
[
{
"id": "sd1adtgsiffc3yt7shder3qeaw",
"type": "import_process",
"priority": 0,
"create_at": 1663619541426,
"start_at": 1663619551774,
"last_activity_at": 1663619551783,
"status": "error",
"progress": -1,
"data": {
"error": "We could not count the users. — ImportProcessWorker: Unable to process import: failed to open file., zip: not a valid zip file",
"import_file": "mattermost-bulk-import.zip"
}
}
]
Step11: root@mattermost ~# zip -T /opt/mattermost/data/import/*.zip
Could not find:
/opt/mattermost/data/import/mattermost-bulk-import.z01
Hit c (change path to where this split file is)
q (abort archive - quit)
or ENTER (try reading this split again):
Step12: root@mattermost ~# zip -T /opt/mattermost/data/import/mattermost-bulk-import.zip
Could not find:
/opt/mattermost/data/import/mattermost-bulk-import.z01
Hit c (change path to where this split file is)
q (abort archive - quit)
or ENTER (try reading this split again):
Step13: root@mattermost ~# cd /opt/mattermost/data/import
root@mattermost .../data/import# unzip mattermost-bulk-import.zip
Archive: mattermost-bulk-import.zip
warning [mattermost-bulk-import.zip]: 462144208 extra bytes at beginning or within zipfile
(attempting to process anyway)
error [mattermost-bulk-import.zip]: start of central directory not found;
zipfile corrupt.
(please check that you have transferred or created the zipfile in the
appropriate BINARY mode and that you have compiled UnZip properly)
Step14: root@mattermost ~# cd /opt/mattermost/data/import/mattermost-bulk-import-and-mattermost-import-jsonl
-bash: cd: /opt/mattermost/data/import/mattermost-bulk-import-and-mattermost-import-jsonl: No such file or directory
root@mattermost ~# cd /opt/mattermost/data/import/mattermost-bulk-import.zip
-bash: cd: /opt/mattermost/data/import/mattermost-bulk-import.zip: Not a directory
root@mattermost ~# zip -T /opt/mattermost/data/import/mattermost-bulk-import.zip
I’m somewhat at a loss here as to what to do further. I am attempting to do all of this on a self hosted mattermost instance rujnning in a turnkeylinux VM on my unraid server (i think i mentioned that before but in case i did not)
Hi again,
just for clarification: You did run the slack-advanced-exporter commands on a windows box, is that right? I’m not sure if there are any limitations by that, but I don’t trust these WSL implementations if that is what you’re using.
You could also run all these commands directly on the Mattermost server - is there a specific reason why you’re not doing that?
You said “When creating the file some come up file share post has missing properties on its file object”: Do you still have the original error message?
If there are any .zXX files additional to the archive, you should see them in the folder C:\Users\Adam\Desktop\mattermost if I interpreted your summary correctly.
My current assumption is that the “Send to compressed folder” on your windows installation either created a split archive and you did not copy all parts, or that this command creates a zip file which is incompatible to the tooling on linux.
For further testing purposes:
If you copy the mattermost-bulk-import.zip file on your windows system to a different folder, f.ex. c:\users\adam\desktop\Test123 and try to extract it in ther using right click, extract to here - does that work?
Yes i did run the slack-advanced-exporter commands on a windows box.
I guess I’m not sure how i would run these commands on the Mattermost server?? I’m just following along with the migration instructions and logically i guess it only made sense to do it on the windows box as that is where my slack export file is saved to and the commands tie back to where that file is saved.
So i wouldnt even know where to begin for how to run those commands on the mattermost server. Would i need to move the slack export over to the mattermost server first and then presumably i could do it?
I can get around windows pretty well and eventually stumble around linux but i dont know it well.
Here is an example of one of the errors:
2022/09/19 07:56:04 SUCCESS COPYING FILE F032SJTSSHG TO DEST bulk-export-attachments/F032SJTSSHG_Screenshot_20220211-142529_Google.jpg
2022/09/19 07:56:04 SUCCESS COPYING FILE F034L68F07N TO DEST bulk-export-attachments/F034L68F07N_Bells Business Plan.docx
2022/09/19 07:56:04 Error retrieving file with id F034WT2H7HP
2022/09/19 07:56:04 SUCCESS COPYING FILE F034BK7KQS2 TO DEST bulk-export-attachments/F034BK7KQS2_Pinnacle Logo - Non Transparent.png
2022/09/19 07:56:04 SUCCESS COPYING FILE F034BKHE7E2 TO DEST bulk-export-attachments/F034BKHE7E2_Pinnacle Logo - Transparent.png
2022/09/19 07:56:04 SUCCESS COPYING FILE F034DCSTCP8 TO DEST bulk-export-attachments/F034DCSTCP8_Letter-of-Intent-1.docx
i do not see any additional .zXX files in that folder.
I do not have an extract here but i do have extract all - if i copy the file to another folder and extract it to that folder it appears to extract perfectly and i am left with the mattermost-import_.jsonl and the “data” folder. Within the data folder i have “bulk-export-attachments” and within there all of the attachments
Ok somewhat trying to figure this out on my own i moved my slack export to the mattermost server:
/opt/mattermost/data/
I then attempted to run the slack-advanced-exporter commands but was returned with
-bash: slack-advanced-exporter: command not found
I then downloaded the linux version of slack-advanced-exporter and moved it over to the mattermost server and extracted the file directly to /opt/mattermost/data/
attempting to run the command still results in the same error noted above.
i then attempted to “cd” into opt/mattermost/data and run the slack-advanced-exporter commands that way to be met with the same error
for anyone following along that runs into this problem – i found the way to run slack-advanced-exporter on the mattermost server the command needs to be sudo ./slack-advanced-exporter
OK, this could be “alright” if the file has been deleted on slack’s end, but I guess you would have to check on that for one specific file to see if it’s really not there anymore or if there’s a problem with the download of the file. If it has been deleted, then this message can be ignored.
With regards to running the commands on the Mattermost server: Yes, you would have to copy the file there (you can do that using tools like WinSCP f.ex.) or if you have a public download link for the slack export, you can also use the command line tool wget to download the archive directly to your Mattermost server.
On linux, for security reasons, binaries in the current working directory cannot be executed directly, so you will have to specify the full path to it (absolute or relative), so if the binary for the slack-advanced-exporter f.ex. lies in your /home/username/slack directory and you’re also in this directory, you can call it with the full path, that is /home/username/slack/slack-advanced-exporter or with the relativ path, that is ./slack-advanced-exporter (. designating the current working directory).
Ok i have been doing this all on the mattermost server this time and its going ok but now i’ve run into an issue doing it on the mattermost server. when i run mmctl import upload
after quite some time i get:
Error: failed to upload data: AppErrorFromJSON: model.utils.decod_json.app_error, body:
Please don’t use the mmctl import upload command when you’re directly on the server. It’s easier to just copy the resulting file over then.
How big is your mattermost-bulk-import.zip file on the server? Is it also 4.6GB in size?
it is 4.42 GiB Ok i moved it over manually and ran the import process on it but yet again i seem to be left with an error
root@mattermost .../mattermost/data# sudo ./mmctl import job show 59565pzifpyhm8at5y5t8kw9fy --json
[
{
"id": "59565pzifpyhm8at5y5t8kw9fy",
"type": "import_process",
"priority": 0,
"create_at": 1663684591968,
"start_at": 1663684598187,
"last_activity_at": 1663684698061,
"status": "error",
"progress": -1,
"data": {
"error": "We could not count the users. — RemoveFile: Unable to remove the file., unable to remove the file import/mattermost-bulk-import.zip: remove data/import/mattermost-bulk-import
.zip: permission denied",
"import_file": "mattermost-bulk-import.zip"
}
}
]
You’re too fast for me 
When moving the data, you need to make sure that the permissions are set correctly, otherwise the system cannot work with the files properly.
Please post the output of:
ls -lh /path/to/your/mattermost/data/import
If there’s only the mattermost-bulk-import.zip file there, then please run the command:
chown mattermost: /path/to/your/mattermost/data/import/*.zip
and try the import again.
Please note that the import will delete the file if it has finished the processing, so if there goes anything wrong with it, it will be gone and you need to create it again, so it’s advisable to keep a copy of this file and do not move it there, but copy it over in case you need to repeat the step (cp vs. mv).
root@mattermost .../mattermost/data# ls -lh /opt/mattermost/data/import
total 4.5G
-rw-r--r-- 1 root root 4.5G Sep 20 13:34 mattermost-bulk-import.zip
after running those commands you stated above i attempted again with the same result - which for some reason does not want to paste here
Also not sure if this matters or not but the “owner” of the file when i look in the webmin is now listed as mattermost:mattermost and I am logged into the shell as root