Attaching files failing using the Bulk Import Tool

Hello pros,
I’m working at importing several hundred Whatsapp messages to Mattermost, along with many attachments. I’m still at the beginning testing stages, as well as writing a program to convert the Whatsapp messages to the proper jsonl format.

I have successfully imported messages, but now on my first test with an attachment, I run in to trouble.
On the first round, the message imported, (and I failed to capture the log lines), and the attachment was there but just as a dead link (no file.) When I deleted the message and ran the import tool again, I see these log lines. One line noting that the file is already uploaded, and then 2 more lines that the file could not be attached. I changed the file name as well in my tests, and watched the logs, and it successfully uploaded, but had the same error lines that it could not attach the file.

Sorry if I’m overlooking something simple, but I’ve gotten myself familiar with the relevant sections of this page, and I’ve searched this forum and the web for similar questions that have been asked, but without success.

Thanks in advance!

Log lines
{"level":"info","ts":1577584224.4083102,"caller":"utils/i18n.go:83","msg":"Loaded system translations","for locale":"en","from locale":"/opt/mattermost/i18n/en.json"}
{"level":"info","ts":1577584224.4086857,"caller":"app/server_app_adapters.go:59","msg":"Server is initializing..."}
{"level":"info","ts":1577584224.41849,"caller":"sqlstore/supplier.go:222","msg":"Pinging SQL","database":"master"}
{"level":"info","ts":1577584224.896113,"caller":"app/license.go:41","msg":"License key from [removed link] required to unlock enterprise features."}
{"level":"info","ts":1577584224.9109914,"caller":"mlog/log.go:166","msg":"Starting up plugins"}
{"level":"error","ts":1577584224.9113975,"caller":"app/plugin.go:145","msg":"Failed to start up plugins","error":"mkdir ./client/plugins: no such file or directory"}
{"level":"info","ts":1577584224.911807,"caller":"app/server.go:216","msg":"Current version is 5.16.0 (5.16.3/Wed Nov  6 14:03:31 UTC 2019/533c7bb3e5847db423387abfcf983e3f947818eb/eb940617df855e3af9c005f9052c7cc3d8e17c30)"}
{"level":"info","ts":1577584224.9123025,"caller":"app/server.go:217","msg":"Enterprise Enabled: true"}
{"level":"info","ts":1577584224.9127293,"caller":"app/server.go:219","msg":"Current working directory is /opt/mattermost/bin"}
{"level":"info","ts":1577584224.913036,"caller":"app/server.go:220","msg":"Loaded config","source":"file:///opt/mattermost/config/config.json"}
{"level":"info","ts":1577584224.9386919,"caller":"mlog/log.go:166","msg":"Starting up plugins"}
{"level":"error","ts":1577584224.9403572,"caller":"app/plugin.go:145","msg":"Failed to start up plugins","error":"mkdir ./client/plugins: no such file or directory"}
{"level":"info","ts":1577584224.946104,"caller":"sqlstore/post_store.go:1356","msg":"Post.Message has size restrictions","max_characters":16383,"max_bytes":65535}
{"level":"info","ts":1577584225.1299665,"caller":"mlog/log.go:166","msg":"Starting up plugins"}
{"level":"error","ts":1577584225.1302803,"caller":"app/plugin.go:145","msg":"Failed to start up plugins","error":"mkdir ./client/plugins: no such file or directory"}
{"level":"error","ts":1577584225.1360729,"caller":"app/server.go:745","msg":"Elasticsearch.start: Elasticsearch is already stopped, "}
{"level":"info","ts":1577584225.1371243,"caller":"app/license.go:41","msg":"License key from [removed link] required to unlock enterprise features."}
{"level":"info","ts":1577584225.1373923,"caller":"mlog/log.go:166","msg":"Starting up plugins"}
{"level":"error","ts":1577584225.1376266,"caller":"app/plugin.go:145","msg":"Failed to start up plugins","error":"mkdir ./client/plugins: no such file or directory"}
Running Bulk Import. This may take a long time.

{"level":"info","ts":1577584225.1542487,"caller":"app/import_functions.go:938","msg":"Skipping uploading of file with name Whatsapp_export/truenorth.jpg, already exists"}
{"level":"error","ts":1577584225.1572797,"caller":"app/import_functions.go:1094","msg":"Error attaching files to post. postId=gfnege4pnb8ymku98a98kfdfqa, fileIds=[y434shi5hbfjpmbs8atetsc7ty dyr64rhqibyz9cccurauzk5keo 97bzawqwribsj8jtd9uapgk9ee], message=SqlFileInfoStore.AttachToPost: Unable to attach the file info to the post, post_id=gfnege4pnb8ymku98a98kfdfqa, file_id=y434shi5hbfjpmbs8atetsc7ty","post_id":"gfnege4pnb8ymku98a98kfdfqa"}
{"level":"error","ts":1577584225.157948,"caller":"app/import_functions.go:1094","msg":"Error attaching files to post. postId=gfnege4pnb8ymku98a98kfdfqa, fileIds=[y434shi5hbfjpmbs8atetsc7ty dyr64rhqibyz9cccurauzk5keo 97bzawqwribsj8jtd9uapgk9ee], message=SqlFileInfoStore.AttachToPost: Unable to attach the file info to the post, post_id=gfnege4pnb8ymku98a98kfdfqa, file_id=dyr64rhqibyz9cccurauzk5keo","post_id":"gfnege4pnb8ymku98a98kfdfqa"}
{"level":"error","ts":1577584225.1585824,"caller":"app/import_functions.go:1094","msg":"Error attaching files to post. postId=gfnege4pnb8ymku98a98kfdfqa, fileIds=[y434shi5hbfjpmbs8atetsc7ty dyr64rhqibyz9cccurauzk5keo 97bzawqwribsj8jtd9uapgk9ee], message=SqlFileInfoStore.AttachToPost: Unable to attach the file info to the post, post_id=gfnege4pnb8ymku98a98kfdfqa, file_id=97bzawqwribsj8jtd9uapgk9ee","post_id":"gfnege4pnb8ymku98a98kfdfqa"}
Finished Bulk Import.
{"level":"info","ts":1577584225.1588776,"caller":"app/server.go:329","msg":"Stopping Server..."}
{"level":"info","ts":1577584225.1590028,"caller":"app/web_hub.go:120","msg":"stopping websocket hub connections"}
{"level":"warn","ts":1577584225.159093,"caller":"app/web_hub.go:125","msg":"We appear to have already sent the stop checking for deadlocks command"}
{"level":"info","ts":1577584225.1608424,"caller":"sqlstore/supplier.go:917","msg":"Closing SqlStore"}
{"level":"info","ts":1577584225.1611612,"caller":"app/server.go:367","msg":"Server stopped"}

An update: I forgot to include the command I’m using to import:

root@mattermost:~# /opt/mattermost/bin/mattermost import bulk test.jsonl --apply

@ginger-cookie have you found a way to attach files via the bulk import tool ?
I spent 5 days on it without success, I still have the same problem !!!

{"type":"version","version":1}
{"type": "post", "post": {"team": "DDDDD", "channel": "DDDDD", "user": "XCFRRTT", "message": "", "create_at": 1587414355000, "replies": [], "attachments": [{"path": "/opt/telegram-to-mattermost/media/1192446106/60/photo_2020-04-20_20-25-55.jpg"}]}}
{"type": "post", "post": {"team": "DDDDD", "channel": "DDDDD", "user": "XCFRRTT", "message": "", "create_at": 1587413829000, "replies": [], "attachments": [{"path": "/opt/telegram-to-mattermost/media/1192446106/59/photo_2020-04-20_20-17-09.jpg"}]}}
{"type": "post", "post": {"team": "DDDDD", "channel": "DDDDD", "user": "XCFRRTT", "message": "", "create_at": 1587413829000, "replies": [], "attachments": [{"path": "/opt/telegram-to-mattermost/media/1192446106/58/photo_2020-04-20_20-17-09.jpg"}]}}
{"type": "post", "post": {"team": "DDDDD", "channel": "DDDDD", "user": "XCFRRTT", "message": "", "create_at": 1587413829000, "replies": [], "attachments": [{"path": "/opt/telegram-to-mattermost/media/1192446106/57/photo_2020-04-20_20-17-09.jpg"}]}}
{"type": "post", "post": {"team": "DDDDD", "channel": "DDDDD", "user": "XCFRRTT", "message": "", "create_at": 1587413829000, "replies": [], "attachments": [{"path": "/opt/telegram-to-mattermost/media/1192446106/56/photo_2020-04-20_20-17-09.jpg"}]}}
{"type": "post", "post": {"team": "DDDDD", "channel": "DDDDD", "user": "XCFRRTT", "message": "", "create_at": 1587413829000, "replies": [], "attachments": [{"path": "/opt/telegram-to-mattermost/media/1192446106/55/photo_2020-04-20_20-17-09.jpg"}]}}

Just Fanstastic,

I found the issue, the bulk command doesn’t set the right rights on /mattermost/data when we are logged as root.

So just set
chown -R mattermo:mattermo /mattermost/data
or
su mattermost
/opt/mattermost/bin/mattermost import bulk test.jsonl --apply

1 Like