Import from JSONL ignores attachments, despite no error messages and "success" at the end

Summary

Importing a JSONL format file with attachments listed in postings silently ignores the attachments with both bin/mattermost import bulk … --apply as well as mmctl import … and both variant report success at the end.

Steps to reproduce

I’ve setup a Mattermost server on Debian 11 Bullseye from the Omnibus packages for Debian. I’ve started IIRC with version 6.4.0 and I’m now at version 6.7.0 (Build Date: Thu May 12 12:44:55 UTC 2022 and Build Hash: 714d065986fa9f6163b417d9760bf707cc237c71). I can’t remember at which version I first tried to import attachments, but

I’ve created already a team and channel (called import-test for the purpose of testing my converter) and my colleagues created their users. I’m only trying to import the history of another chat via bulk import — no user creation, no team creation, no channel creation.

So far my tool telegram2mm already successfully imports years of history of a Telegram group into Mattermost — except for any attachments, be it pictures, PDF files or other files. (JFTR: The code which is used for attachments is not yet checked in, because it doesn’t work yet as described in this posting. If anyone is interested, I can push an incomplete WIP feature branch or two.)

Except for attachements, my tool successfully uses a generated .jsonl file from a Telegram export and imports it into Mattermost using these steps:

  • mmctl import upload … --json
  • mmctl import list available --json
  • mmctl import process … --json
  • mmctl import job show … --json

The last command is repeated until it neither returns pending nor in_progress. The last mmctl import job show always returned success for me. (Unless I had an issue unrelated to this one of course, but those seem all fixed. :grin:)

For testing purposes I’ve faked a minimal Telegram group chat export which results in the following mattermost_import,jsonl file included in the uploaded .zip file:

{"type":"version","version":1}
{"attachments":[{"path":"/home/myuser/telegram2mm/ChatExport_2022-05-10/photos/photo_1@28-12-2019_04-09-26.jpg"}],"post":{"channel":"import-test","create_at":1653944037000,"message":"Grüsse vom 36C3 😃","team":"myteam","user":"abe"},"props":{"attachments":[{}]},"type":"post"}

The team, channel, user and file referenced in the JSONL exist and the file is readable for the mattermost user:

mattermost@chat:/opt/mattermost$ ls -l /home/myuser/telegram2mm/ChatExport_2022-05-10/photos/photo_1@28-12-2019_04-09-26.jpg
-rw-r--r-- 1 myuser myteam 252258 May 10 15:04 /home/myuser/telegram2mm/ChatExport_2022-05-10/photos/photo_1@28-12-2019_04-09-26.jpg

I’ve also tried these variants:

  • Used relative instead of absolute paths.
  • Used relative instead of absolute paths and tried to put all the attachments into the ZIP file, too. Depending on the amount of attachment files, it didn’t change anything, or, with all attachment files, resulted in an error message saying that the uploaded ZIP file is too big. (With attachments it was slightly above 400 MB in the end.)
  • Tried it without the "props":{"attachments":[{}]} part. (Actually that was my first try.)
  • Tried it with every / in the path being escaped as \/ as the used JSON library does by default. (I actually added code to remove that escaping to see if it makes a difference.
  • Changed the code to use cd /opt/mattermost; bin/mattermost import bulk /tmp/some_temporary_directory/mattermost_import.jsonl --apply (checked first vith --validate, too.)
  • Ran the latter two commands manually on the commandline instead via my tool.

*Some more details

Output from the bin/mattermost import bulk mattermost_import.jsonl --apply command when running man:

mattermost@chat:/home/myuser/telegram2mm/telegram2mm$ ./telegram2mm.pl ../import_config.yml ../ChatExport_2022-05-10/result_minimal_attachment_test.json
{"timestamp":"2022-05-31 00:44:43.938 Z","level":"info","msg":"Server is initializing...","caller":"app/server.go:262","go_version":"go1.18.1"}
{"timestamp":"2022-05-31 00:44:43.939 Z","level":"info","msg":"Pinging SQL","caller":"sqlstore/store.go:226","database":"master"}
{"timestamp":"2022-05-31 00:44:43.968 Z","level":"debug","msg":"Deleting any unused pre-release features","caller":"sqlstore/preference_store.go:25"}
{"timestamp":"2022-05-31 00:44:43.983 Z","level":"debug","msg":"We could not find the license key in the database or on disk at","caller":"utils/license.go:114","filename":"/opt/mattermost/config/mattermost.mattermost-license"}
{"timestamp":"2022-05-31 00:44:43.984 Z","level":"error","msg":"License key from https://mattermost.com required to unlock enterprise features.","caller":"app/license.go:147","error":"resource: License id: "}
{"timestamp":"2022-05-31 00:44:43.984 Z","level":"info","msg":"Starting websocket hubs","caller":"app/web_hub.go:93","number_of_hubs":4}
{"timestamp":"2022-05-31 00:44:43.984 Z","level":"info","msg":"Loaded system translations","caller":"i18n/i18n.go:93","for locale":"en","from locale":"/opt/mattermost/i18n/en.json"}
{"timestamp":"2022-05-31 00:44:43.991 Z","level":"debug","msg":"Hub is starting","caller":"app/web_hub.go:411","index":0}
{"timestamp":"2022-05-31 00:44:43.991 Z","level":"debug","msg":"Hub is starting","caller":"app/web_hub.go:411","index":1}
{"timestamp":"2022-05-31 00:44:43.991 Z","level":"debug","msg":"Hub is starting","caller":"app/web_hub.go:411","index":2}
{"timestamp":"2022-05-31 00:44:43.991 Z","level":"debug","msg":"Hub is starting","caller":"app/web_hub.go:411","index":3}
{"timestamp":"2022-05-31 00:44:44.004 Z","level":"info","msg":"Current version is 6.7.0 (6.7.0/Thu May 12 12:44:55 UTC 2022/714d065986fa9f6163b417d9760bf707cc237c71/ba596678562c7f9ef1a51bfa7b4dcbb8e5c0e937)","caller":"app/server.go:566","current_version":"6.7.0","build_number":"6.7.0","build_date":"Thu May 12 12:44:55 UTC 2022","build_hash":"714d065986fa9f6163b417d9760bf707cc237c71","build_hash_enterprise":"ba596678562c7f9ef1a51bfa7b4dcbb8e5c0e937"}
{"timestamp":"2022-05-31 00:44:44.004 Z","level":"info","msg":"Enterprise Build","caller":"app/server.go:575","enterprise_build":true}
{"timestamp":"2022-05-31 00:44:44.004 Z","level":"info","msg":"Printing current working","caller":"app/server.go:581","directory":"/opt/mattermost"}
{"timestamp":"2022-05-31 00:44:44.004 Z","level":"info","msg":"Loaded config","caller":"app/server.go:582","source":"postgres://mmuser:@localhost:5432/mattermost?sslmode=disable&connect_timeout=10"}
{"timestamp":"2022-05-31 00:44:44.005 Z","level":"debug","msg":"Logging metrics enabled","caller":"app/server.go:948"}
{"timestamp":"2022-05-31 00:44:44.005 Z","level":"debug","msg":"Will fetch notices from","caller":"app/product_notices.go:344","url":"https://notices.mattermost.com/","skip_cache":false}
{"timestamp":"2022-05-31 00:44:44.031 Z","level":"debug","msg":"We could not find the license key in the database or on disk at","caller":"utils/license.go:114","filename":"/opt/mattermost/config/mattermost.mattermost-license"}
{"timestamp":"2022-05-31 00:44:44.031 Z","level":"error","msg":"License key from https://mattermost.com required to unlock enterprise features.","caller":"app/license.go:147","error":"resource: License id: "}
{"timestamp":"2022-05-31 00:44:44.032 Z","level":"debug","msg":"Logging metrics enabled","caller":"app/server.go:948"}
{"timestamp":"2022-05-31 00:44:44.032 Z","level":"info","msg":"Starting up plugins","caller":"app/plugin.go:185"}
{"timestamp":"2022-05-31 00:44:44.032 Z","level":"debug","msg":"Enabling plugin health check job","caller":"plugin/environment.go:512","interval_s":"30s"}
{"timestamp":"2022-05-31 00:44:44.032 Z","level":"info","msg":"Syncing plugins from the file store","caller":"app/plugin.go:259"}
{"timestamp":"2022-05-31 00:44:44.033 Z","level":"debug","msg":"Plugin health check job starting.","caller":"plugin/health_check.go:31"}
{"timestamp":"2022-05-31 00:44:44.033 Z","level":"debug","msg":"Removing local installation of managed plugin before sync","caller":"app/plugin.go:283","plugin_id":"com.mattermost.plugin-channel-export"}
{"timestamp":"2022-05-31 00:44:44.033 Z","level":"debug","msg":"Removing local installation of managed plugin before sync","caller":"app/plugin.go:283","plugin_id":"com.mattermost.apps"}
{"timestamp":"2022-05-31 00:44:44.042 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-antivirus-v0.1.2-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.047 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-nps-v1.2.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.060 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-apps-v1.0.1-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.080 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-github-v2.0.1-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.100 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/focalboard-v0.15.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.120 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-autolink-v1.2.2-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.132 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-channel-export-v1.0.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.147 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-jenkins-v1.1.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.167 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-welcomebot-v1.2.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.188 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-custom-attributes-v1.3.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.204 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-jira-v2.4.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.219 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-playbooks-v1.27.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.239 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-zoom-v1.5.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.267 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-aws-SNS-v1.2.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.285 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-calls-v0.4.9-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.301 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-gitlab-v1.3.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:45.785 Z","level":"debug","msg":"Installing prepackaged plugin","caller":"app/plugin.go:949","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-channel-export-v1.0.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:45.924 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.plugin-channel-export","wrapped_extras":"pathplugins/com.mattermost.plugin-channel-export/server/dist/plugin-linux-amd64args[plugins/com.mattermost.plugin-channel-export/server/dist/plugin-linux-amd64]"}
{"timestamp":"2022-05-31 00:44:45.927 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.plugin-channel-export","wrapped_extras":"pathplugins/com.mattermost.plugin-channel-export/server/dist/plugin-linux-amd64pid1378367"}
{"timestamp":"2022-05-31 00:44:45.927 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.plugin-channel-export","wrapped_extras":"pathplugins/com.mattermost.plugin-channel-export/server/dist/plugin-linux-amd64"}
{"timestamp":"2022-05-31 00:44:45.982 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.plugin-channel-export","wrapped_extras":"version1"}
{"timestamp":"2022-05-31 00:44:45.982 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.plugin-channel-export","wrapped_extras":"address/tmp/plugin344088630networkunixtimestamp2022-05-31T00:44:45.959Z"}
{"timestamp":"2022-05-31 00:44:46.032 Z","level":"debug","msg":"Installing prepackaged plugin","caller":"app/plugin.go:949","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-apps-v1.0.1-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:46.191 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"pathplugins/com.mattermost.apps/server/dist/plugin-linux-amd64args[plugins/com.mattermost.apps/server/dist/plugin-linux-amd64]"}
{"timestamp":"2022-05-31 00:44:46.191 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"pathplugins/com.mattermost.apps/server/dist/plugin-linux-amd64pid1378374"}
{"timestamp":"2022-05-31 00:44:46.191 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"pathplugins/com.mattermost.apps/server/dist/plugin-linux-amd64"}
{"timestamp":"2022-05-31 00:44:46.218 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"version1"}
{"timestamp":"2022-05-31 00:44:46.218 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"address/tmp/plugin1869240709networkunixtimestamp2022-05-31T00:44:46.218Z"}
{"timestamp":"2022-05-31 00:44:46.264 Z","level":"info","msg":"Failed to fetch license twice. May incorrectly default to on-prem mode.","caller":"app/plugin_api.go:937","plugin_id":"com.mattermost.apps"}
{"timestamp":"2022-05-31 00:44:46.264 Z","level":"debug","msg":"OnActivate: Version: 1.0.1, [9e0ed4b](https://github.com/mattermost/mattermost-plugin-apps/commit/9e0ed4b738ae265eb45fa34e9fee950e1360362d), built Wed 06 Apr 2022 09:59:18 PM UTC, Cloud Mode: false, Developer Mode: false, Allow install over HTTP: true","caller":"app/plugin_api.go:934","plugin_id":"com.mattermost.apps"}
{"timestamp":"2022-05-31 00:44:46.265 Z","level":"debug","msg":"New AWS client","caller":"app/plugin_api.go:934","plugin_id":"com.mattermost.apps","purpose":"Manifest store","region":"us-east-1","access":"<nil>","secret":"<nil>"}
{"timestamp":"2022-05-31 00:44:46.265 Z","level":"debug","msg":"Initialized persistent store","caller":"app/plugin_api.go:934","plugin_id":"com.mattermost.apps"}
{"timestamp":"2022-05-31 00:44:46.265 Z","level":"debug","msg":"Initialized \"HTTP\" upstream.","caller":"app/plugin_api.go:934","plugin_id":"com.mattermost.apps"}
{"timestamp":"2022-05-31 00:44:46.266 Z","level":"debug","msg":"Skipped \"AWS Lambda\" upstream: not configured.","caller":"app/plugin_api.go:934","plugin_id":"com.mattermost.apps","error":"AWS credentials are not set: not found"}
{"timestamp":"2022-05-31 00:44:46.266 Z","level":"debug","msg":"Initialized \"Mattermost Plugin\" upstream.","caller":"app/plugin_api.go:934","plugin_id":"com.mattermost.apps"}
{"timestamp":"2022-05-31 00:44:46.266 Z","level":"debug","msg":"Skipped \"OpenFaaS\" upstream: not configured.","caller":"app/plugin_api.go:934","plugin_id":"com.mattermost.apps","error":"OPENFAAS_URL environment variable must be defined: not found"}
{"timestamp":"2022-05-31 00:44:46.266 Z","level":"debug","msg":"Initialized the app proxy","caller":"app/plugin_api.go:934","plugin_id":"com.mattermost.apps"}
{"timestamp":"2022-05-31 00:44:46.267 Z","level":"info","msg":"Plugin activated","caller":"app/plugin_api.go:937","plugin_id":"com.mattermost.apps"}
{"timestamp":"2022-05-31 00:44:46.273 Z","level":"debug","msg":"Not auto installing/upgrade because plugin was disabled","caller":"app/plugin.go:970","plugin_id":"playbooks","version":""}
{"timestamp":"2022-05-31 00:44:46.273 Z","level":"debug","msg":"Not auto installing/upgrade because plugin was disabled","caller":"app/plugin.go:970","plugin_id":"focalboard","version":""}
Running Bulk Import. This may take a long time.

{"timestamp":"2022-05-31 00:44:46.290 Z","level":"info","msg":"Post.Message has size restrictions","caller":"sqlstore/post_store.go:2359","max_characters":16383,"max_bytes":65535}
Finished Bulk Import.
{"timestamp":"2022-05-31 00:44:46.296 Z","level":"info","msg":"Stopping Server...","caller":"app/server.go:976"}
{"timestamp":"2022-05-31 00:44:46.296 Z","level":"info","msg":"stopping websocket hub connections","caller":"app/web_hub.go:113"}
{"timestamp":"2022-05-31 00:44:46.297 Z","level":"info","msg":"Server stopped","caller":"app/server.go:1056"}
{"timestamp":"2022-05-31 00:44:46.297 Z","level":"info","msg":"Shutting down plugins","caller":"app/plugin.go:339"}
{"timestamp":"2022-05-31 00:44:46.297 Z","level":"debug","msg":"Disabling plugin health check job","caller":"plugin/environment.go:521"}
{"timestamp":"2022-05-31 00:44:46.306 Z","level":"debug","msg":"plugin process exited","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"pathplugins/com.mattermost.apps/server/dist/plugin-linux-amd64pid1378374"}
{"timestamp":"2022-05-31 00:44:46.306 Z","level":"debug","msg":"plugin exited","caller":"plugin/hclog_adapter.go:54","plugin_id":"com.mattermost.apps"}
{"timestamp":"2022-05-31 00:44:46.306 Z","level":"debug","msg":"plugin process exited","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.plugin-channel-export","wrapped_extras":"pathplugins/com.mattermost.plugin-channel-export/server/dist/plugin-linux-amd64pid1378367"}
{"timestamp":"2022-05-31 00:44:46.306 Z","level":"debug","msg":"plugin exited","caller":"plugin/hclog_adapter.go:54","plugin_id":"com.mattermost.plugin-channel-export"}

I noticed the warning {"timestamp":"2022-05-31 00:44:46.290 Z","level":"info","msg":"Post.Message has size restrictions","caller":"sqlstore/post_store.go:2359","max_characters":16383,"max_bytes":65535} in the output, but the file in question is only 364 bytes short.

What else I’ve done

  • I’ve verified that file uploading is enabled and works. I’ve checked the latter by posting a picture via the web frontend into the channel into which I’m trying to import. Worked.

  • I’ve exported that channel and looked at the resulting export file. It though looks very different to what is documented as import format.

  • To be on the safe side, I also changed MM_PLUGINSETTINGS_ENABLEUPLOADS respectively enable_plugin_uploads to true in /etc/mattermost/mmomni.mattermost.env and /etc/mattermost/mmomni.yml. But I’m not sure if this is about uploading via plugins or about uploading plugins.

Documentation I’ve read:

Other resources I’ve looked at in this forum:

I’ve also looked at these other converter and importer tools:

Expected behavior

I would expected that either the included picture or other files are shown (or at least offered for download) inside the chat, or an error message at import time is thrown. But neither happens.

Observed behavior

Postings with attachments show up in the according channel as if they never had one, i.e. only show the included text or smiley if any.

There isn’t any trace of the attachment. Both, pictures as well as other files are neither shown nor offered for download.

Note

I would have added a picture of how the postings look like without attachment despite they should have one to this posting, but uploading the picture to this forum reproducibly fails for me at the end. The JavaScript console says:

[Uppy] [03:24:12] Failed to upload mattermost.png AwsS3/Multipart: Could not read the ETag header. This likely means CORS is not configured correctly on the S3 Bucket. See https://uppy.io/docs/aws-s3-multipart#S3-Bucket-Configuration for instructions.

(JFTR: This last error message is not related to my attachment import error described in this posting but just an issue with this forum.)

Hi @xtaran and welcome to the Mattermost forums!

Sorry for the late reply, but I just recently was digging a bit deeper into the slack importer and while doing some searching I stumbled upon your post.
I’m not 100% sure about the syntax for the generic mattermost import format, but comparing your data structure for the attachment with the one mattermost creates using their mmetl tool, the format differs and this might also help you with finding out why your attachments cannot be imported (if this is still the case):

Your JSON:


  "attachments": [
    {
      "path": "/home/myuser/telegram2mm/ChatExport_2022-05-10/photos/photo_1@28-12-2019_04-09-26.jpg"
    }
  ],
  "post": {
    "channel": "import-test",
    "create_at": 1653944037000,
    "message": "Grüsse vom 36C3 😃",
    "team": "myteam",
    "user": "abe"
  },
  "props": {
    "attachments": [
      {}
    ]
  },
  "type": "post"
}

Mattermost’s converter slack attachment in JSON:

{
  "type": "post",
  "post": {
    "team": "slackimport",
    "channel": "testchannel2",
    "user": "support",
    "message": "",
    "props": null,
    "create_at": 1660276097841,
    "replies": [],
    "attachments": [
      {
        "path": "bulk-export-attachments/F03TALS0D2R_klugscheisser.PNG"
      }
    ]
  }
}

The latter one definitely works, maybe that helps!

Also, would you consider porting parts of your tool into mmetl? I think this is a great resource for getting more import modules together in just one point and not having to worry about the basics every source platform needs for migrating to the mattermost format.

1 Like

Thanks a lot for a first reply on this topic. As you might have seen on how late I reply back, I had mostly given up on this topic (and hence also on Mattermost itself).

I’m not 100% sure about the syntax for the generic mattermost import format, but comparing your data structure for the attachment with the one mattermost creates using their mmetl tool.

I’m aware of their mmetl tool, but as far as I could see, it just an intermediate step for imports from Slack.

the format differs and this might also help you with finding out why your attachments cannot be imported

Thanks for that idea! I’m not super confident that these formats are closely related to each other, but I’ve tried to move the attachments array one level deeper inside the JSON as you’ve suggested anyway as I’m still out of ideas otherwise.

This though resulted in this not so nice error message:

Checking status for job ID d7nzbmw4c3ratg1nxo77gicjmc: pending
"mmctl import job show d7nzbmw4c3ratg1nxo77gicjmc" exited with  256 and this output:

Error: AppErrorFromJSON: model.utils.decode_json.app_error, body: <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>

And in /var/log/mattermost/mattermost.log the last line is:

{"timestamp":"2022-10-06 13:14:17.445 Z","level":"info","msg":"Post.Message has size restrictions","caller":"sqlstore/post_store.go:2442","max_characters":16383,"max_bytes":65535}

(Current Mattermost version is 7.3.0.)

So it seems that this unfortunately is a kinda “more wrong” way. Probably because everything inside the post hash goes into the SQL database while the attachements itself don’t. Then again this probably also means that the attachment path in some way has been expanded.

Which kinda prompts me to do another dig into the Mattermost (server) source code. At least I now have some error messages with paths to look for in the code! :crazy_face:

Also, would you consider porting parts of your tool into mmetl?

No, it would mean to start from scratch again despite I’m already very far — everything works as expected except attachments. And I first would have to learn Go. And trying to get things done is IMHO not the right place to learn a new programming language.

Looking at the test suite example imports at https://github.com/mattermost/mattermost-server/blob/5e69c6b02f604ce1aecb4c22cc58f9412cc685a2/app/import_test.go#L184-L185 @agriesser seems to be right though with regards to the data structure: attachments is a key inside the post hash.

So I now have to figure out what actually causes this error message.

cc @noxer


1 Like

Actually, mmetl strives to be a data transformation tool for various inputs and right now, only slack has been implemented. The idea behind that is that for the generic import to work, a generic output format needs to be created and although different messaging software uses different input formats, they all need to be converted to one common output format for Mattermost to be able to import it.

There’s a limit on the maximum size for a single post and some tools support bigger post sizes (f.ex. slack does). So what needs to be done here is to split such longer posts into multiple posts (ideally within a thread), but that requires some fiddling. For testing purposes, you could just trim the input to let’s say 12k characters. If the import works with that, you can tackle the message splitting later on.

1 Like

Turned out that this is just the last message after boot up of the Mattermost daemon and (as declared) just informational.

What actually happened is that now the Mattermost daemon crashes upon my import try and systemd restarts it after a few seconds which results in like 20 lines of log entries.

Here’s what I try to import:

[…]egram2mm/telegram2mm → acat /tmp/telegram2mm_TfI3lYKIr.zip mattermost_import.jsonl
{"type":"version","version":1}
{"post":{"attachments":[{"path":"\/home\/username\/telegram2mm\/telegram2mm\/..\/ChatExport\/photos\/photo_1@28-12-2019_04-09-26.jpg"}],"channel":"import-test","create_at":1665092757000,"message":"Grüsse vom 36C3 😃","team":"it-sec","user":"abe"},"type":"post"}

(The file exists and is readable by the user mattermost.)

Here’s what I digged out of app/import_test.go as two examples with attachments:

{"type": "version", "version": 1}
{"type": "post", "post": {"team": "teamname", "channel": "channelname", "user": "username", "message": "Hello World", "create_at": 123456789012, "attachments":[{"path": "/path/to/test.png"}]}}
{"type": "post", "post": {"team": "teamname", "channel": "channelname", "user": "username3", "message": "Hey Everyone!", "create_at": 123456789013, "attachments":[{"path": "/path/to/test.png"}]}}

And I really don’t see a relevant difference.

Still at Mattermost 7.3.0. (Installed via the provided .deb packages from the APT repo.)

Actually it never was an issue. It’s just an informational message at boot up. I got confused because this is always the last log entry at boot up and since I crashed the server, it’s always the last log message I saw after import tries.

My test message is 303 Bytes long in JSONL format. Telegram has a limit of 4k per message anyways, too, so that should be way below those 16kB. I just wasn’t sure about the images, but I looked into the DB itself and there’s only a tiny preview image and the real image is actually on disk under the non-standard branch /var/opt/.

So this message was actually a red herring. But the crash causing that message in the end is what scares me.

Yah, I know this message on startup - just thought you got it after your import, that’s why I wanted to mention it.
The attributes props and replies are missing in your import, not sure if they’re required, but you can try to add them with empty values as shown in my example above to rule that out.
Another thing you could try is an empty message and just the attachment, only to see if that makes a difference and I’m not sure if the importer will support absolute paths, I’m pretty sure it doesn’t, because when you upload the file to the server, it needs to be relative to the extracted zip file contents (below the data folder as already mentioned). Can you try to fix that first?

1 Like

If so, then the documentation needs to be fixed. Because the path attribute in the example post object at Bulk loading data — Mattermost documentation clearly sports a leading slash.

Thanks for that detail! I was looking for that information for quite a while in the documentation and never found it. It would be great to add it to Bulk loading data — Mattermost documentation

Will try, thanks!

Hrm, does that mean that I first have to copy those files into the /var/opt/mattermost/data/ tree myself? I assumed that the Mattermost server actually doers that when it parses these paths from the JSONL file in the uploaded ZIP archive.

This is not mentioned in the documentation at Bulk loading data — Mattermost documentation either. (Actually I even tried to put the attachment files into the ZIP file as well, but that quickly failed for a real-world data import due to rather low size limits.)

In general the bulk import documentation lacks any details about how the paths given for attachments are actually handled or where they are meant to point to. :face_with_diagonal_mouth:

Anyway, I will try to create a new, probably world-writable directory under /var/opt/mattermost/data/ (maybe named incoming or upload or so), and copy the all the files there.

My understanding so far of how this works is that you’re supposed to run the mmctl, mmetl and other tools on your local desktop and then use the mmctl import tool to actually upload the file to the server. What happens then is that the server will put the uploaded zip file (which needs to be in the specified format) into a tempoary directory (usually the data directory) and when you then process the import, it will extract the zip file from there to some (other?) temporary directory and will work with the jsonl files and the attachments from there, then. At least this is what I observed when troubleshooting serveral user’s slack imports (but this is not an official information here, I’m just writing down what I experienced).

You can, if you’re working on the Mattermost server itself or if you transfer the zip file to it by other means (f.ex. SFTP), skip the upload process and manually copy the import file to the data directory then. You should then see it listed with the mmctl import list available command (and btw, you can also use the --local flag for mmctl if you enabled the local socket connection in your config.json if you’re working directly on the Mattermost server).

If you want to use the Mattermost bulk import utility (which is what I definitely would suggest), then your export tool needs to create a ZIP file that looks like this:

# zipinfo mattermost-bulk-import.zip
Archive:  mattermost-bulk-import.zip
Zip file size: 11485125 bytes, number of entries: 5
drwxr-xr-x  3.0 unx        0 bx stor 22-Aug-22 12:33 data/
drwxr-xr-x  3.0 unx        0 bx stor 22-Aug-12 06:08 data/bulk-export-attachments/
-rw-r--r--  3.0 unx   568424 bx defN 22-Aug-12 06:08 data/bulk-export-attachments/F03TALS0D2R_klugscheisser.PNG
-rw-r--r--  3.0 unx 11470200 bx defN 22-Aug-12 06:08 data/bulk-export-attachments/F03TS8P6LE5_WinSCP-5.21.2-Setup.exe
-rw-r--r--  3.0 unx     3498 tx defN 22-Aug-12 11:33 mattermost_import.jsonl
5 files, 12042122 bytes uncompressed, 11484105 bytes compressed:  4.6%

The bulk-export-attachments folder is not necessary, but the data folder is. In your jsonl file you then need to reference relative to the data directory, in my case, it would be:

{
  "type": "post",
  "post": {
    "team": "slackimport",
    "channel": "testchannel2",
    "user": "support",
    "message": "",
    "props": null,
    "create_at": 1660276097841,
    "replies": [],
    "attachments": [
      {
        "path": "bulk-export-attachments/F03TALS0D2R_klugscheisser.PNG"
      }
    ]
  }
}

Oh, and BTW, thanks for the documentation pointers - I think it would make sense to go over that after you’re up and running here. The docs team is always happy to receive hints like that :slight_smile:

1 Like

Thanks for the reply. Took a while until I found time again to work on this issue.

Luckily Mattermost 7.5.1 no more crashes :tada: on my minimal JSONL import example, but instead even gave me a helpful (!) error message :tada::tada::

{"timestamp":"2022-12-02 16:08:29.779 Z","level":"warn","msg":"Error while processing import attachments. Objects might be broken.","caller":"app/import.go:206","error":"attachment \"data/data/photos/photo_1@28-12-2019_04-09-26.jpg\" not found in map"}
{"timestamp":"2022-12-02 16:08:29.785 Z","level":"warn","msg":"failed to import attachment","caller":"app/import_functions.go:1546","path":"data/data/photos/photo_1@28-12-2019_04-09-26.jpg","error":"BulkImport: Error reading the file at: \"data/data/photos/photo_1@28-12-2019_04-09-26.jpg\", open data/data/photos/photo_1@28-12-2019_04-09-26.jpg: no such file or directory"}

And suddenly everything was clear:

  • As @agriesser hinted, the files need to go into a data subdirectory.
  • What was not obvious before that error message thrown since 7.5.1 was, that I must not refer to that data subdirectory inside the mattermost_import.jsonl file but just to the subdirectories beneath it. It became obvious when I saw in the error message that Mattermost tried to access something starting with data/data/…

So instead of

[{"path":"data\/photos\/photo_1@28-12-2019_04-09-26.jpg"}],"channel":"import-test","create_at":1665092757000,"message":"Grüsse vom 36C3 😃","team":"it-sec","user":"abe"},"type":"post"}

I just had to use

[{"path":"photos\/photo_1@28-12-2019_04-09-26.jpg"}],"channel":"import-test","create_at":1665092757000,"message":"Grüsse vom 36C3 😃","team":"it-sec","user":"abe"},"type":"post"}

Thanks to all those who helped to get this fixed in 7.5.x and especially @agriesser!

[Edit one day later] Additional note: I think that this pull request actually fixed my crash issue:

I suspect that my erroneous path to the attachment triggered an exception which wanted to be logged, but ran into that null pointer exception while doing so which then caused the crash.

2 Likes

Thanks for the wrap-up, greatly appreciated :slight_smile: and glad to hear you’re up and running now.

1 Like