Import from JSONL ignores attachments, despite no error messages and "success" at the end

Summary

Importing a JSONL format file with attachments listed in postings silently ignores the attachments with both bin/mattermost import bulk … --apply as well as mmctl import … and both variant report success at the end.

Steps to reproduce

I’ve setup a Mattermost server on Debian 11 Bullseye from the Omnibus packages for Debian. I’ve started IIRC with version 6.4.0 and I’m now at version 6.7.0 (Build Date: Thu May 12 12:44:55 UTC 2022 and Build Hash: 714d065986fa9f6163b417d9760bf707cc237c71). I can’t remember at which version I first tried to import attachments, but

I’ve created already a team and channel (called import-test for the purpose of testing my converter) and my colleagues created their users. I’m only trying to import the history of another chat via bulk import — no user creation, no team creation, no channel creation.

So far my tool telegram2mm already successfully imports years of history of a Telegram group into Mattermost — except for any attachments, be it pictures, PDF files or other files. (JFTR: The code which is used for attachments is not yet checked in, because it doesn’t work yet as described in this posting. If anyone is interested, I can push an incomplete WIP feature branch or two.)

Except for attachements, my tool successfully uses a generated .jsonl file from a Telegram export and imports it into Mattermost using these steps:

  • mmctl import upload … --json
  • mmctl import list available --json
  • mmctl import process … --json
  • mmctl import job show … --json

The last command is repeated until it neither returns pending nor in_progress. The last mmctl import job show always returned success for me. (Unless I had an issue unrelated to this one of course, but those seem all fixed. :grin:)

For testing purposes I’ve faked a minimal Telegram group chat export which results in the following mattermost_import,jsonl file included in the uploaded .zip file:

{"type":"version","version":1}
{"attachments":[{"path":"/home/myuser/telegram2mm/ChatExport_2022-05-10/photos/photo_1@28-12-2019_04-09-26.jpg"}],"post":{"channel":"import-test","create_at":1653944037000,"message":"Grüsse vom 36C3 😃","team":"myteam","user":"abe"},"props":{"attachments":[{}]},"type":"post"}

The team, channel, user and file referenced in the JSONL exist and the file is readable for the mattermost user:

mattermost@chat:/opt/mattermost$ ls -l /home/myuser/telegram2mm/ChatExport_2022-05-10/photos/photo_1@28-12-2019_04-09-26.jpg
-rw-r--r-- 1 myuser myteam 252258 May 10 15:04 /home/myuser/telegram2mm/ChatExport_2022-05-10/photos/photo_1@28-12-2019_04-09-26.jpg

I’ve also tried these variants:

  • Used relative instead of absolute paths.
  • Used relative instead of absolute paths and tried to put all the attachments into the ZIP file, too. Depending on the amount of attachment files, it didn’t change anything, or, with all attachment files, resulted in an error message saying that the uploaded ZIP file is too big. (With attachments it was slightly above 400 MB in the end.)
  • Tried it without the "props":{"attachments":[{}]} part. (Actually that was my first try.)
  • Tried it with every / in the path being escaped as \/ as the used JSON library does by default. (I actually added code to remove that escaping to see if it makes a difference.
  • Changed the code to use cd /opt/mattermost; bin/mattermost import bulk /tmp/some_temporary_directory/mattermost_import.jsonl --apply (checked first vith --validate, too.)
  • Ran the latter two commands manually on the commandline instead via my tool.

*Some more details

Output from the bin/mattermost import bulk mattermost_import.jsonl --apply command when running man:

mattermost@chat:/home/myuser/telegram2mm/telegram2mm$ ./telegram2mm.pl ../import_config.yml ../ChatExport_2022-05-10/result_minimal_attachment_test.json
{"timestamp":"2022-05-31 00:44:43.938 Z","level":"info","msg":"Server is initializing...","caller":"app/server.go:262","go_version":"go1.18.1"}
{"timestamp":"2022-05-31 00:44:43.939 Z","level":"info","msg":"Pinging SQL","caller":"sqlstore/store.go:226","database":"master"}
{"timestamp":"2022-05-31 00:44:43.968 Z","level":"debug","msg":"Deleting any unused pre-release features","caller":"sqlstore/preference_store.go:25"}
{"timestamp":"2022-05-31 00:44:43.983 Z","level":"debug","msg":"We could not find the license key in the database or on disk at","caller":"utils/license.go:114","filename":"/opt/mattermost/config/mattermost.mattermost-license"}
{"timestamp":"2022-05-31 00:44:43.984 Z","level":"error","msg":"License key from https://mattermost.com required to unlock enterprise features.","caller":"app/license.go:147","error":"resource: License id: "}
{"timestamp":"2022-05-31 00:44:43.984 Z","level":"info","msg":"Starting websocket hubs","caller":"app/web_hub.go:93","number_of_hubs":4}
{"timestamp":"2022-05-31 00:44:43.984 Z","level":"info","msg":"Loaded system translations","caller":"i18n/i18n.go:93","for locale":"en","from locale":"/opt/mattermost/i18n/en.json"}
{"timestamp":"2022-05-31 00:44:43.991 Z","level":"debug","msg":"Hub is starting","caller":"app/web_hub.go:411","index":0}
{"timestamp":"2022-05-31 00:44:43.991 Z","level":"debug","msg":"Hub is starting","caller":"app/web_hub.go:411","index":1}
{"timestamp":"2022-05-31 00:44:43.991 Z","level":"debug","msg":"Hub is starting","caller":"app/web_hub.go:411","index":2}
{"timestamp":"2022-05-31 00:44:43.991 Z","level":"debug","msg":"Hub is starting","caller":"app/web_hub.go:411","index":3}
{"timestamp":"2022-05-31 00:44:44.004 Z","level":"info","msg":"Current version is 6.7.0 (6.7.0/Thu May 12 12:44:55 UTC 2022/714d065986fa9f6163b417d9760bf707cc237c71/ba596678562c7f9ef1a51bfa7b4dcbb8e5c0e937)","caller":"app/server.go:566","current_version":"6.7.0","build_number":"6.7.0","build_date":"Thu May 12 12:44:55 UTC 2022","build_hash":"714d065986fa9f6163b417d9760bf707cc237c71","build_hash_enterprise":"ba596678562c7f9ef1a51bfa7b4dcbb8e5c0e937"}
{"timestamp":"2022-05-31 00:44:44.004 Z","level":"info","msg":"Enterprise Build","caller":"app/server.go:575","enterprise_build":true}
{"timestamp":"2022-05-31 00:44:44.004 Z","level":"info","msg":"Printing current working","caller":"app/server.go:581","directory":"/opt/mattermost"}
{"timestamp":"2022-05-31 00:44:44.004 Z","level":"info","msg":"Loaded config","caller":"app/server.go:582","source":"postgres://mmuser:@localhost:5432/mattermost?sslmode=disable&connect_timeout=10"}
{"timestamp":"2022-05-31 00:44:44.005 Z","level":"debug","msg":"Logging metrics enabled","caller":"app/server.go:948"}
{"timestamp":"2022-05-31 00:44:44.005 Z","level":"debug","msg":"Will fetch notices from","caller":"app/product_notices.go:344","url":"https://notices.mattermost.com/","skip_cache":false}
{"timestamp":"2022-05-31 00:44:44.031 Z","level":"debug","msg":"We could not find the license key in the database or on disk at","caller":"utils/license.go:114","filename":"/opt/mattermost/config/mattermost.mattermost-license"}
{"timestamp":"2022-05-31 00:44:44.031 Z","level":"error","msg":"License key from https://mattermost.com required to unlock enterprise features.","caller":"app/license.go:147","error":"resource: License id: "}
{"timestamp":"2022-05-31 00:44:44.032 Z","level":"debug","msg":"Logging metrics enabled","caller":"app/server.go:948"}
{"timestamp":"2022-05-31 00:44:44.032 Z","level":"info","msg":"Starting up plugins","caller":"app/plugin.go:185"}
{"timestamp":"2022-05-31 00:44:44.032 Z","level":"debug","msg":"Enabling plugin health check job","caller":"plugin/environment.go:512","interval_s":"30s"}
{"timestamp":"2022-05-31 00:44:44.032 Z","level":"info","msg":"Syncing plugins from the file store","caller":"app/plugin.go:259"}
{"timestamp":"2022-05-31 00:44:44.033 Z","level":"debug","msg":"Plugin health check job starting.","caller":"plugin/health_check.go:31"}
{"timestamp":"2022-05-31 00:44:44.033 Z","level":"debug","msg":"Removing local installation of managed plugin before sync","caller":"app/plugin.go:283","plugin_id":"com.mattermost.plugin-channel-export"}
{"timestamp":"2022-05-31 00:44:44.033 Z","level":"debug","msg":"Removing local installation of managed plugin before sync","caller":"app/plugin.go:283","plugin_id":"com.mattermost.apps"}
{"timestamp":"2022-05-31 00:44:44.042 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-antivirus-v0.1.2-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.047 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-nps-v1.2.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.060 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-apps-v1.0.1-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.080 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-github-v2.0.1-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.100 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/focalboard-v0.15.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.120 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-autolink-v1.2.2-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.132 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-channel-export-v1.0.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.147 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-jenkins-v1.1.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.167 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-welcomebot-v1.2.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.188 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-custom-attributes-v1.3.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.204 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-jira-v2.4.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.219 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-playbooks-v1.27.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.239 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-zoom-v1.5.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.267 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-aws-SNS-v1.2.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.285 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-calls-v0.4.9-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:44.301 Z","level":"debug","msg":"Processing prepackaged plugin","caller":"app/plugin.go:919","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-gitlab-v1.3.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:45.785 Z","level":"debug","msg":"Installing prepackaged plugin","caller":"app/plugin.go:949","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-channel-export-v1.0.0-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:45.924 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.plugin-channel-export","wrapped_extras":"pathplugins/com.mattermost.plugin-channel-export/server/dist/plugin-linux-amd64args[plugins/com.mattermost.plugin-channel-export/server/dist/plugin-linux-amd64]"}
{"timestamp":"2022-05-31 00:44:45.927 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.plugin-channel-export","wrapped_extras":"pathplugins/com.mattermost.plugin-channel-export/server/dist/plugin-linux-amd64pid1378367"}
{"timestamp":"2022-05-31 00:44:45.927 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.plugin-channel-export","wrapped_extras":"pathplugins/com.mattermost.plugin-channel-export/server/dist/plugin-linux-amd64"}
{"timestamp":"2022-05-31 00:44:45.982 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.plugin-channel-export","wrapped_extras":"version1"}
{"timestamp":"2022-05-31 00:44:45.982 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.plugin-channel-export","wrapped_extras":"address/tmp/plugin344088630networkunixtimestamp2022-05-31T00:44:45.959Z"}
{"timestamp":"2022-05-31 00:44:46.032 Z","level":"debug","msg":"Installing prepackaged plugin","caller":"app/plugin.go:949","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-apps-v1.0.1-linux-amd64.tar.gz"}
{"timestamp":"2022-05-31 00:44:46.191 Z","level":"debug","msg":"starting plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"pathplugins/com.mattermost.apps/server/dist/plugin-linux-amd64args[plugins/com.mattermost.apps/server/dist/plugin-linux-amd64]"}
{"timestamp":"2022-05-31 00:44:46.191 Z","level":"debug","msg":"plugin started","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"pathplugins/com.mattermost.apps/server/dist/plugin-linux-amd64pid1378374"}
{"timestamp":"2022-05-31 00:44:46.191 Z","level":"debug","msg":"waiting for RPC address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"pathplugins/com.mattermost.apps/server/dist/plugin-linux-amd64"}
{"timestamp":"2022-05-31 00:44:46.218 Z","level":"debug","msg":"using plugin","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"version1"}
{"timestamp":"2022-05-31 00:44:46.218 Z","level":"debug","msg":"plugin address","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"address/tmp/plugin1869240709networkunixtimestamp2022-05-31T00:44:46.218Z"}
{"timestamp":"2022-05-31 00:44:46.264 Z","level":"info","msg":"Failed to fetch license twice. May incorrectly default to on-prem mode.","caller":"app/plugin_api.go:937","plugin_id":"com.mattermost.apps"}
{"timestamp":"2022-05-31 00:44:46.264 Z","level":"debug","msg":"OnActivate: Version: 1.0.1, [9e0ed4b](https://github.com/mattermost/mattermost-plugin-apps/commit/9e0ed4b738ae265eb45fa34e9fee950e1360362d), built Wed 06 Apr 2022 09:59:18 PM UTC, Cloud Mode: false, Developer Mode: false, Allow install over HTTP: true","caller":"app/plugin_api.go:934","plugin_id":"com.mattermost.apps"}
{"timestamp":"2022-05-31 00:44:46.265 Z","level":"debug","msg":"New AWS client","caller":"app/plugin_api.go:934","plugin_id":"com.mattermost.apps","purpose":"Manifest store","region":"us-east-1","access":"<nil>","secret":"<nil>"}
{"timestamp":"2022-05-31 00:44:46.265 Z","level":"debug","msg":"Initialized persistent store","caller":"app/plugin_api.go:934","plugin_id":"com.mattermost.apps"}
{"timestamp":"2022-05-31 00:44:46.265 Z","level":"debug","msg":"Initialized \"HTTP\" upstream.","caller":"app/plugin_api.go:934","plugin_id":"com.mattermost.apps"}
{"timestamp":"2022-05-31 00:44:46.266 Z","level":"debug","msg":"Skipped \"AWS Lambda\" upstream: not configured.","caller":"app/plugin_api.go:934","plugin_id":"com.mattermost.apps","error":"AWS credentials are not set: not found"}
{"timestamp":"2022-05-31 00:44:46.266 Z","level":"debug","msg":"Initialized \"Mattermost Plugin\" upstream.","caller":"app/plugin_api.go:934","plugin_id":"com.mattermost.apps"}
{"timestamp":"2022-05-31 00:44:46.266 Z","level":"debug","msg":"Skipped \"OpenFaaS\" upstream: not configured.","caller":"app/plugin_api.go:934","plugin_id":"com.mattermost.apps","error":"OPENFAAS_URL environment variable must be defined: not found"}
{"timestamp":"2022-05-31 00:44:46.266 Z","level":"debug","msg":"Initialized the app proxy","caller":"app/plugin_api.go:934","plugin_id":"com.mattermost.apps"}
{"timestamp":"2022-05-31 00:44:46.267 Z","level":"info","msg":"Plugin activated","caller":"app/plugin_api.go:937","plugin_id":"com.mattermost.apps"}
{"timestamp":"2022-05-31 00:44:46.273 Z","level":"debug","msg":"Not auto installing/upgrade because plugin was disabled","caller":"app/plugin.go:970","plugin_id":"playbooks","version":""}
{"timestamp":"2022-05-31 00:44:46.273 Z","level":"debug","msg":"Not auto installing/upgrade because plugin was disabled","caller":"app/plugin.go:970","plugin_id":"focalboard","version":""}
Running Bulk Import. This may take a long time.

{"timestamp":"2022-05-31 00:44:46.290 Z","level":"info","msg":"Post.Message has size restrictions","caller":"sqlstore/post_store.go:2359","max_characters":16383,"max_bytes":65535}
Finished Bulk Import.
{"timestamp":"2022-05-31 00:44:46.296 Z","level":"info","msg":"Stopping Server...","caller":"app/server.go:976"}
{"timestamp":"2022-05-31 00:44:46.296 Z","level":"info","msg":"stopping websocket hub connections","caller":"app/web_hub.go:113"}
{"timestamp":"2022-05-31 00:44:46.297 Z","level":"info","msg":"Server stopped","caller":"app/server.go:1056"}
{"timestamp":"2022-05-31 00:44:46.297 Z","level":"info","msg":"Shutting down plugins","caller":"app/plugin.go:339"}
{"timestamp":"2022-05-31 00:44:46.297 Z","level":"debug","msg":"Disabling plugin health check job","caller":"plugin/environment.go:521"}
{"timestamp":"2022-05-31 00:44:46.306 Z","level":"debug","msg":"plugin process exited","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.apps","wrapped_extras":"pathplugins/com.mattermost.apps/server/dist/plugin-linux-amd64pid1378374"}
{"timestamp":"2022-05-31 00:44:46.306 Z","level":"debug","msg":"plugin exited","caller":"plugin/hclog_adapter.go:54","plugin_id":"com.mattermost.apps"}
{"timestamp":"2022-05-31 00:44:46.306 Z","level":"debug","msg":"plugin process exited","caller":"plugin/hclog_adapter.go:52","plugin_id":"com.mattermost.plugin-channel-export","wrapped_extras":"pathplugins/com.mattermost.plugin-channel-export/server/dist/plugin-linux-amd64pid1378367"}
{"timestamp":"2022-05-31 00:44:46.306 Z","level":"debug","msg":"plugin exited","caller":"plugin/hclog_adapter.go:54","plugin_id":"com.mattermost.plugin-channel-export"}

I noticed the warning {"timestamp":"2022-05-31 00:44:46.290 Z","level":"info","msg":"Post.Message has size restrictions","caller":"sqlstore/post_store.go:2359","max_characters":16383,"max_bytes":65535} in the output, but the file in question is only 364 bytes short.

What else I’ve done

  • I’ve verified that file uploading is enabled and works. I’ve checked the latter by posting a picture via the web frontend into the channel into which I’m trying to import. Worked.

  • I’ve exported that channel and looked at the resulting export file. It though looks very different to what is documented as import format.

  • To be on the safe side, I also changed MM_PLUGINSETTINGS_ENABLEUPLOADS respectively enable_plugin_uploads to true in /etc/mattermost/mmomni.mattermost.env and /etc/mattermost/mmomni.yml. But I’m not sure if this is about uploading via plugins or about uploading plugins.

Documentation I’ve read:

Other resources I’ve looked at in this forum:

I’ve also looked at these other converter and importer tools:

Expected behavior

I would expected that either the included picture or other files are shown (or at least offered for download) inside the chat, or an error message at import time is thrown. But neither happens.

Observed behavior

Postings with attachments show up in the according channel as if they never had one, i.e. only show the included text or smiley if any.

There isn’t any trace of the attachment. Both, pictures as well as other files are neither shown nor offered for download.

Note

I would have added a picture of how the postings look like without attachment despite they should have one to this posting, but uploading the picture to this forum reproducibly fails for me at the end. The JavaScript console says:

[Uppy] [03:24:12] Failed to upload mattermost.png AwsS3/Multipart: Could not read the ETag header. This likely means CORS is not configured correctly on the S3 Bucket. See https://uppy.io/docs/aws-s3-multipart#S3-Bucket-Configuration for instructions.

(JFTR: This last error message is not related to my attachment import error described in this posting but just an issue with this forum.)