No attachments in mattermost after import of Slack Advanced Exporter file via CLI or mmetl

Summary
All post attachments (images/pdfs/mp4) are missing after import to mattermost

Steps to reproduce
https://docs.mattermost.com/administration/migrating.html#migrating-from-slack-using-the-mattermost-cli
Download Slack export from Slack.
Download attachments via Slack Advanced Exporter. The export-with-attachments.zip has all the files in the __uploads folder.

sudo ./slack-advanced-exporter-linux-amd64 --input-archive slack.zip --output-archive export-with-attachments.zip fetch-attachments

Import resulting archive to Mattermost 5.24.2 using CLI tool

/opt/mattermost/bin/mattermost import slack testteam /root/export-with-attachments.zip

After performing the above steps I verified the newly imported team. All the messages are in place, except for attachments. Those were completely missing.

Debian 10
Mattermost 5.24.2

Expected behavior
The mattermost channel has all the attachments like the slack channel.

Mattermost Slack Import Log
#######
{"level":"info","ts":1597277647.6629498,"caller":"utils/i18n.go:83","msg":"Loaded system translations","for locale":"en","from locale":"/opt/mattermost/i18n/en.json"}
{"level":"info","ts":1597277647.6632302,"caller":"app/server_app_adapters.go:58","msg":"Server is initializing..."}
{"level":"info","ts":1597277647.6673126,"caller":"sqlstore/supplier.go:227","msg":"Pinging SQL","database":"master"}
{"level":"error","ts":1597277647.7305863,"caller":"app/server_app_adapters.go:152","msg":"Mail server connection test is failed: SendEmailNotifications is not true"}
{"level":"info","ts":1597277647.7317753,"caller":"app/license.go:37","msg":"License key from https://mattermost.com required to unlock enterprise features."}
{"level":"info","ts":1597277647.7370763,"caller":"mlog/log.go:167","msg":"Starting up plugins"}
{"level":"error","ts":1597277647.737134,"caller":"app/plugin.go:161","msg":"Failed to start up plugins","error":"mkdir ./client/plugins: no such file or directory"}
{"level":"info","ts":1597277647.7371817,"caller":"app/server.go:273","msg":"Current version is 5.24.0 (5.24.2/Fri Jun 26 19:10:02 UTC 2020/2c394d78ca036a439aa06fedd6ad187da535d5fc/8239891909042ef8e47e2b81b5d515ec65d8938c)","current_version":"5.24.0","build_number":"5.24.2","build_date":"Fri Jun 26 19:10:02 UTC 2020","build_hash":"2c394d78ca036a439aa06fedd6ad187da535d5fc","build_hash_enterprise":"8239891909042ef8e47e2b81b5d515ec65d8938c"}
{"level":"info","ts":1597277647.7372105,"caller":"app/server.go:282","msg":"Enterprise Build","enterprise_build":true}
{"level":"info","ts":1597277647.7372365,"caller":"app/server.go:288","msg":"Printing current working","directory":"/root"}
{"level":"info","ts":1597277647.7372603,"caller":"app/server.go:289","msg":"Loaded config","source":"file:///opt/mattermost/config/config.json"}
{"level":"info","ts":1597277647.7566736,"caller":"sqlstore/post_store.go:1571","msg":"Post.Message has size restrictions","max_characters":16383,"max_bytes":65535}
{"level":"info","ts":1597277647.7570834,"caller":"mlog/log.go:167","msg":"Starting up plugins"}
{"level":"error","ts":1597277647.7571416,"caller":"app/plugin.go:161","msg":"Failed to start up plugins","error":"mkdir ./client/plugins: no such file or directory"}
{"level":"info","ts":1597277647.7919083,"caller":"mlog/log.go:167","msg":"Starting up plugins"}
{"level":"error","ts":1597277647.791985,"caller":"app/plugin.go:161","msg":"Failed to start up plugins","error":"mkdir ./client/plugins: no such file or directory"}
{"level":"info","ts":1597277647.7944965,"caller":"bleveengine/bleve.go:267","msg":"UpdateConf Bleve"}
{"level":"info","ts":1597277647.795125,"caller":"app/license.go:37","msg":"License key from https://mattermost.com required to unlock enterprise features."}
{"level":"info","ts":1597277647.7955935,"caller":"mlog/log.go:167","msg":"Starting up plugins"}
{"level":"error","ts":1597277647.7956364,"caller":"app/plugin.go:161","msg":"Failed to start up plugins","error":"mkdir ./client/plugins: no such file or directory"}
Running Slack Import. This may take a long time for large teams or teams with many messages.
{"level":"info","ts":1597277648.0550404,"caller":"app/admin.go:151","msg":"Purging all caches"}
{"level":"info","ts":1597277648.05632,"caller":"app/license.go:37","msg":"License key from https://mattermost.com required to unlock enterprise features."}

Mattermost Slack Import Log

Users created:
===============

Slack user merged with an existing Mattermost user with matching email *** and username ***.
The Integration/Slack Bot user with email slackimportuser_xdbe9txcrbr3bk58og196ze4te@localhost and password *** has been imported.

Channels added:
=================

zufällig
slackexport
testbilder2
allgemein

Notes:
=======

- Some messages may not have been imported because they were not supported by this importer.
- Slack bot messages are currently not supported.
- Additional errors may be found in the server logs.


Finished Slack Import.

{"level":"info","ts":1597277648.0575478,"caller":"app/server.go:416","msg":"Stopping Server..."}
{"level":"info","ts":1597277648.05758,"caller":"app/web_hub.go:130","msg":"stopping websocket hub connections"}
{"level":"info","ts":1597277648.0584583,"caller":"app/server.go:469","msg":"Server stopped"}

I have exactly the same phenomenon. Mattermost 5.25.2 on Debian 10, both using the CLI import method as well as the bulk import using mmetl. No difference.
The Slack export archive is rather big (2,4 GB), not sure if that is an issue?

Hello @SvenS I’m sorry to hear that, does your error log also have the same information such as was provided by @netteschnecke in the original post, as seen below?

No, something else - both the --validate and the --apply start with these

{"level":"info","ts":1606398571.003683,"caller":"utils/i18n.go:83","msg":"Loaded system translations","for locale":"en","from locale":"/opt/mattermost/i18n/en.json"}
{"level":"info","ts":1606398571.0038996,"caller":"app/server_app_adapters.go:58","msg":"Server is initializing..."}
{"level":"info","ts":1606398571.0068812,"caller":"sqlstore/supplier.go:227","msg":"Pinging SQL","database":"master"}
{"level":"info","ts":1606398571.1148207,"caller":"mlog/log.go:167","msg":"Starting up plugins"}
{"level":"info","ts":1606398571.1149483,"caller":"app/plugin.go:202","msg":"Syncing plugins from the file store"}
{"level":"error","ts":1606398571.553881,"caller":"app/plugin.go:793","msg":"Failed to install prepackaged plugin","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-nps-v1.0.4-linux-amd64.tar.gz","error":"Failed to install extracted prepackaged plugin /opt/mattermost/prepackaged_plugins/mattermost-plugin-nps-v1.0.4-linux-amd64.tar.gz: installExtractedPlugin: Unable to move plugin from temporary directory to final destination. Another plugin may be using the same directory name., destination already exists","errorVerbose":"installExtractedPlugin: Unable to move plugin from temporary directory to final destination. Another plugin may be using the same directory name., destination already exists\nFailed to install extracted prepackaged plugin /opt/mattermost/prepackaged_plugins/mattermost-plugin-nps-v1.0.4-linux-amd64.tar.gz\ngithub.com/mattermost/mattermost-server/v5/app.(*App).processPrepackagedPlugin\n\tgithub.com/mattermost/mattermost-server/v5/app/plugin.go:845\ngithub.com/mattermost/mattermost-server/v5/app.(*App).processPrepackagedPlugins.func2\n\tgithub.com/mattermost/mattermost-server/v5/app/plugin.go:791\nruntime.goexit\n\truntime/asm_amd64.s:1373"}
{"level":"info","ts":1606398571.5779762,"caller":"app/server.go:273","msg":"Current version is 5.25.0 (5.25.2/Fri Jul 31 12:28:34 UTC 2020/7c21fabef8a03c0e130da73cd8419f263c876e32/none)","current_version":"5.25.0","build_number":"5.25.2","build_date":"Fri Jul 31 12:28:34 UTC 2020","build_hash":"7c21fabef8a03c0e130da73cd8419f263c876e32","build_hash_enterprise":"none"}

This is followed by many of these:

{"level":"error","ts":1606398579.6887288,"caller":"app/import_functions.go:1361","msg":"Error attaching files to post.","post_id":"k6wr8detnpyj7euaxp48sbx47y","post_file_ids":["4f8oabacctro5cd9obmf1b143o"],"error":"SqlFileInfoStore.AttachToPost: Unable to attach the file info to the post., post_id=k6wr8detnpyj7euaxp48sbx47y, file_id=4f8oabacctro5cd9obmf1b143o"}

I noticed something in your error message that is throwing quite a bit of a red flag out and really caught my attention. Specifically, this:

errorVerbose":"installExtractedPlugin: Unable to move plugin from temporary directory to final destination. Another plugin may be using the same directory name., destination already exists\nFailed to install extracted prepackaged plugin /opt/mattermost/prepackaged_plugins/mattermost-plugin-nps-v1.0.4-linux-amd64.tar.gz\ngithub.com/mattermost/mattermost-server/v5/app.(*App).processPrepackagedPlugin\n\tgithub.com/mattermost/mattermost-server/v5/app/plugin.go:845\ngithub.com/mattermost/mattermost-server/v5/app.(*App).processPrepackagedPlugins.func2\n\tgithub.com/mattermost/mattermost-server/v5/app/plugin.go:791\nruntime.goexit\n\truntime/asm_amd64.s:1373"}

What really stood out to me what the error that a file already existed, when it seems like it shouldn’t be the case.

Would you be willing to run the command (from your Linux command line)
ls -alh /opt/mattermost/prepackaged_plugins/ and copy/paste the output here? I’m just curious to see if there is indeed a file with the same name already existing.

Sure, here you are:

-rw-r--r--  1 mattermost mattermost 8.7M Jul 31 14:31 mattermost-plugin-antivirus-v0.1.2-linux-amd64.tar.gz
-rw-r--r--  1 mattermost mattermost  566 Jul 31 14:31 mattermost-plugin-antivirus-v0.1.2-linux-amd64.tar.gz.sig
-rw-r--r--  1 mattermost mattermost 8.7M Jul 31 14:31 mattermost-plugin-autolink-v1.1.2-linux-amd64.tar.gz
-rw-r--r--  1 mattermost mattermost  566 Jul 31 14:31 mattermost-plugin-autolink-v1.1.2-linux-amd64.tar.gz.sig
-rw-r--r--  1 mattermost mattermost 8.8M Jul 31 14:31 mattermost-plugin-aws-SNS-v1.0.2-linux-amd64.tar.gz
-rw-r--r--  1 mattermost mattermost  566 Jul 31 14:31 mattermost-plugin-aws-SNS-v1.0.2-linux-amd64.tar.gz.sig
-rw-r--r--  1 mattermost mattermost 8.8M Jul 31 14:31 mattermost-plugin-custom-attributes-v1.2.0-linux-amd64.tar.gz
-rw-r--r--  1 mattermost mattermost  566 Jul 31 14:31 mattermost-plugin-custom-attributes-v1.2.0-linux-amd64.tar.gz.sig
-rw-r--r--  1 mattermost mattermost 9.9M Jul 31 14:31 mattermost-plugin-github-v0.14.0-linux-amd64.tar.gz
-rw-r--r--  1 mattermost mattermost  566 Jul 31 14:31 mattermost-plugin-github-v0.14.0-linux-amd64.tar.gz.sig
-rw-r--r--  1 mattermost mattermost 9.5M Jul 31 14:31 mattermost-plugin-gitlab-v1.1.0-linux-amd64.tar.gz
-rw-r--r--  1 mattermost mattermost  566 Jul 31 14:31 mattermost-plugin-gitlab-v1.1.0-linux-amd64.tar.gz.sig
-rw-r--r--  1 mattermost mattermost 9.0M Jul 31 14:31 mattermost-plugin-jenkins-v1.0.0-linux-amd64.tar.gz
-rw-r--r--  1 mattermost mattermost  566 Jul 31 14:31 mattermost-plugin-jenkins-v1.0.0-linux-amd64.tar.gz.sig
-rw-r--r--  1 mattermost mattermost  10M Jul 31 14:31 mattermost-plugin-jira-v2.3.2-linux-amd64.tar.gz
-rw-r--r--  1 mattermost mattermost  566 Jul 31 14:31 mattermost-plugin-jira-v2.3.2-linux-amd64.tar.gz.sig
-rw-r--r--  1 mattermost mattermost 8.9M Jul 31 14:31 mattermost-plugin-nps-v1.0.4-linux-amd64.tar.gz
-rw-r--r--  1 mattermost mattermost  566 Jul 31 14:31 mattermost-plugin-nps-v1.0.4-linux-amd64.tar.gz.sig
-rw-r--r--  1 mattermost mattermost 8.6M Jul 31 14:31 mattermost-plugin-welcomebot-v1.1.1-linux-amd64.tar.gz
-rw-r--r--  1 mattermost mattermost  566 Jul 31 14:31 mattermost-plugin-welcomebot-v1.1.1-linux-amd64.tar.gz.sig
-rw-r--r--  1 mattermost mattermost 8.9M Jul 31 14:31 mattermost-plugin-zoom-v1.3.1-linux-amd64.tar.gz
-rw-r--r--  1 mattermost mattermost  566 Jul 31 14:31 mattermost-plugin-zoom-v1.3.1-linux-amd64.tar.gz.sig

Hello! Thank you for providing that information, it’s quite useful!

It looks to me like this may be caused by the file already existing in the directory, as I found it here:

I’m thinking (just a thought) that if you were to remove these files, and attempt to do the same functions that have been giving you errors, that it might help. You can remove the two files with the following commands (for linux command line):
sudo rm /opt/mattermost/prepackaged_plugins/mattermost-plugin-nps-v1.0.4-linux-amd64.tar.gz
and
sudo rm /opt/mattermost/prepackaged_plugins/mattermost-plugin-nps-v1.0.4-linux-amd64.tar.gz.sig

Let me know if it works!

1 Like

Thanks a lot, that worked!. Well, mostly. A lot of the attachments are now there, but not all of them. When using mmetl, I get the best results - but I also noticed that after the conversion, not all attachments that are present in the slack export are also in the bulk-export-attachments folder. When I compare the .json files in the slack archive of files that work and files that do not, I unfortunately cannot find any differences. Some of them are rather long and complex names, others are rather short, all of them are .jpg - and none of them work. All of them are in the same channel, so it must have something to do with that - but I cannot figure it out.

Could you perhaps set the Mattermost logging level to DEBUG and reimport the data, and see if there are any importing errors logged?

Tried that, but didn’t really yield anything new - the attachments that are still missing get lost during the mmetl conversion. I can’t find anything in that log regarding the missing files. The logged errors are about slack related channels and bot messages, which cannot be converted - which is fine, don’t need them. However, nothing about the missing attachments. They are in the slack export but not in the bulk-attachments folder after the mmetl conversion. Everything in there gets imported into mattermost just fine, but for some reason mmetl does not take everything without logging any message about it.

Are you on the latest Mattermost version? Just a thought that occured to me…

No, on 5.25.2. Next Upgrade is planned for January. But since the problem occurs when doing the conversion with mmectl, I thought that mattermost itself is not involved at that early point?

I’m not sure if it is or not, @ahmaddanial would you perhaps be able to provide more insight into when mattermost itself begins to handle slack imports?

Ok, updating to the most recent version to Mattermost also did not change anything regarding this issue. Anything new from your side? Or is this simply not solvable?

Hello, @SvenS

I came across this - add support for new slack post format and multiple attachments in slack importer by svelle · Pull Request #11415 · mattermost/mattermost-server · GitHub which discusses the ability to import attachments from Slack to Mattermost as well. Still a work in progress based on the status of the PR.

Perhaps you would like to chime into the discussion as well to share the importance of having this implemented soon.

Thanks! Just read through it, maybe I do not understand it completely, but it seems to be completed resp. already merged. I seem to remember the issue back then, I also had a slack-import that didn’t work at all until this fix came along.
Also, in my case the direct import of the advanced-slack-exporter result into mattermost using mattermost import slack teamname /locationOfExportfile fails to import a lot of the attachments, only if I convert the export first using mmetl continued by Bulk-import into Mattermost I get most of them - but not all. The get lost during the mmetl conversion, as they are no longer present in the resulting bulk-export-attachments folder. I simply have no idea at all why. The missing attachments are from different users, have different sizes and file types. I cannot recognize any pattern. No error messages in the conversion log.

1 Like

Hi, @SvenS

Got it. Thanks for the clarification as I must have gotten a different understanding on the actual issue, which only applies to attachments. Apologies on that.

I will reach out to the team internally and verify if we have bug report created for this issue and get back to you once I get an answer.

1 Like

Hi,

any news on this? Or is it simply not solvable?

Hi, @SvenS

No news from the team since the last time I reached out to them. I am following up on this again and will keep you updated if I hear anything on the failed attachments import from Slack.

1 Like

Hi,

I’m also having issue with importing Slack attachments.

To me it seems like the problem happens during the transformation with mmetl transform slack ... as the output file doesn’t have the attachments.
This process has some errors like Error retrieving file with id ABCD, is it because it needs to be authenticated to fetch the files?

Thanks for your help,
Philippe

1 Like