I am trying to migrate data from cloud to self-hosted server by this manual Mattermost workspace migration — Mattermost documentation
All is working until the last point. I run “mmctl extract run”, created extract job, and that’s all… The job is always in progress status.
Hi @VasilyBelan and welcome to the Mattermost forums!
The job most likely crashed, can you see anything relevant in the logs? (/opt/mattermost/logs/mattermost.log by default)
Also, what Mattermost server version are you trying to import to?
Also I found a error messages
{“timestamp”:“2023-06-08 20:13:09.598 +05:00”,“level”:“error”,“msg”:“License key from https://mattermost.com required to unlock enterprise features.”,“caller”:“platform/license.go:101”,“error”:“resource: License id: “}
{“timestamp”:“2023-06-08 20:13:09.652 +05:00”,“level”:“info”,“msg”:“Starting up plugins”,“caller”:“app/plugin.go:218”}
{“timestamp”:“2023-06-08 20:13:09.652 +05:00”,“level”:“info”,“msg”:“Syncing plugins from the file store”,“caller”:“app/plugin.go:299”}
{“timestamp”:“2023-06-08 20:13:12.690 +05:00”,“level”:“info”,“msg”:“Ensuring Feedbackbot exists”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“com.mattermost.nps”}
{“timestamp”:“2023-06-08 20:13:13.650 +05:00”,“level”:“info”,“msg”:“failed to load initial configuration”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“com.mattermost.apps”,“error”:“plugin requires Mattermost Site URL to be set”,“plugin_caller”:“server/plugin.go:85”}
{“timestamp”:“2023-06-08 20:13:13.651 +05:00”,“level”:“info”,“msg”:“plugin process exited”,“caller”:“plugin/hclog_adapter.go:61”,“plugin_id”:“com.mattermost.apps”,“wrapped_extras”:“pathplugins/com.mattermost.apps/server/dist/plugin-linux-amd64pid1878”}
{“timestamp”:“2023-06-08 20:13:13.655 +05:00”,“level”:“error”,“msg”:“Failed to install prepackaged plugin”,“caller”:“app/plugin.go:964”,“path”:”/opt/mattermost/prepackaged_plugins/mattermost-plugin-apps-v1.2.1-linux-amd64.tar.gz”,“error”:“Failed to install extracted prepackaged plugin /opt/mattermost/prepackaged_pl>
{“timestamp”:“2023-06-08 20:13:16.599 +05:00”,“level”:“warn”,“msg”:“Rudder credentials are not set. Disabling analytics.”,“caller”:“app/plugin_api.go:979”,“plugin_id”:“playbooks”,“plugin_caller”:“github.com/mattermost/mattermost-plugin-playbooks/server/plugin.go:141”}
{“timestamp”:“2023-06-08 20:13:16.699 +05:00”,“level”:“info”,“msg”:“registered collection and topic type”,“caller”:“app/collection.go:33”,“plugin_id”:“playbooks”,“collection_type”:“run”,“topic_type”:“status”}
{“timestamp”:“2023-06-08 20:13:16.700 +05:00”,“level”:“info”,“msg”:“registered collection and topic type”,“caller”:“app/collection.go:33”,“plugin_id”:“playbooks”,“collection_type”:“run”,“topic_type”:“task”}
{“timestamp”:“2023-06-08 20:13:23.534 +05:00”,“level”:“info”,“msg”:“connectDatabase”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“focalboard”,“dbType”:“postgres”}
{“timestamp”:“2023-06-08 20:13:23.555 +05:00”,“level”:“info”,“msg”:“Schema migration table is correct format”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“focalboard”}
{“timestamp”:“2023-06-08 20:13:23.556 +05:00”,“level”:“info”,“msg”:”{"level":"info","msg":"Pinging SQL","fields":{"database":"master"}}\n",“caller”:“io/io.go:429”,“plugin_id”:“focalboard”,“source”:“plugin_stderr”}
{“timestamp”:“2023-06-08 20:13:23.635 +05:00”,“level”:“info”,“msg”:“Initialized notification backend”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“focalboard”,“name”:“notifyMentions”}
{“timestamp”:“2023-06-08 20:13:23.635 +05:00”,“level”:“info”,“msg”:“Initialized notification backend”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“focalboard”,“name”:“notifySubscriptions”}
{“timestamp”:“2023-06-08 20:13:23.635 +05:00”,“level”:“info”,“msg”:“Initialized notification backend”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“focalboard”,“name”:“notifyLogger”}
{“timestamp”:“2023-06-08 20:13:23.642 +05:00”,“level”:“info”,“msg”:“Focalboard server”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“focalboard”,“version”:“7.10.0”,“edition”:“plugin”,“build_number”:“4612623265”,“build_date”:“"Tue Apr 4 21:49:19 UTC 2023"”,“build_hash”:“55ba9d8624ac1359239e3c0c5f4223a88e78a7df”}
{“timestamp”:“2023-06-08 20:13:23.642 +05:00”,“level”:“info”,“msg”:“Server.Start”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“focalboard”}
{“timestamp”:“2023-06-08 20:13:23.646 +05:00”,“level”:“info”,“msg”:“Boards product successfully started.”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“focalboard”}
{“timestamp”:“2023-06-08 20:13:23.718 +05:00”,“level”:“info”,“msg”:“got public IP address”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“com.mattermost.calls”,“origin”:“main.(*logger).Info log.go:95”,“addr”:“92.50.185.130”}
{“timestamp”:“2023-06-08 20:13:23.718 +05:00”,“level”:“info”,“msg”:“rtc: server is listening on udp :8443”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“com.mattermost.calls”,“origin”:“main.(*logger).Info log.go:95”}
{“timestamp”:“2023-06-08 20:13:23.718 +05:00”,“level”:“info”,“msg”:“rtc: server is listening on udp :8443”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“com.mattermost.calls”,“origin”:“main.(*logger).Info log.go:95”}
{“timestamp”:“2023-06-08 20:13:24.004 +05:00”,“level”:“info”,“msg”:“failed to load initial configuration”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“com.mattermost.apps”,“error”:“plugin requires Mattermost Site URL to be set”,“plugin_caller”:“server/plugin.go:85”}
{“timestamp”:“2023-06-08 20:13:24.005 +05:00”,“level”:“info”,“msg”:“plugin process exited”,“caller”:“plugin/hclog_adapter.go:61”,“plugin_id”:“com.mattermost.apps”,“wrapped_extras”:“pathplugins/com.mattermost.apps/server/dist/plugin-linux-amd64pid1910”}
{“timestamp”:“2023-06-08 20:13:24.005 +05:00”,“level”:“error”,“msg”:“Unable to activate plugin”,“caller”:“app/plugin.go:171”,“plugin_id”:“com.mattermost.apps”,“error”:“failed to load initial configuration: plugin requires Mattermost Site URL to be set”}
{“timestamp”:“2023-06-08 20:13:24.019 +05:00”,“level”:“info”,“msg”:“Starting Server…”,“caller”:“app/server.go:899”}
{“timestamp”:“2023-06-08 20:13:24.019 +05:00”,“level”:“info”,“msg”:“Server is listening on [::]:8065”,“caller”:“app/server.go:971”,“address”:“[::]:8065”}
{“timestamp”:“2023-06-08 20:13:24.019 +05:00”,“level”:“info”,“msg”:“Sending systemd READY notification.”,“caller”:“commands/server.go:130”}
{“timestamp”:“2023-06-08 20:13:24.755 +05:00”,“level”:“warn”,“msg”:“unable to extract file content”,“caller”:“docextractor/combine.go:35”,“error”:“exec: "pdftotext": executable file not found in $PATH”}
{“timestamp”:“2023-06-08 20:13:24.755 +05:00”,“level”:“warn”,“msg”:“unable to extract file content”,“caller”:“docextractor/combine.go:35”,“error”:“error copying data into temporary file: write /tmp/pdflib173887679: copy_file_range: use of closed file”}
{“timestamp”:“2023-06-08 20:13:24.755 +05:00”,“level”:“warn”,“msg”:“unable to extract file content”,“caller”:“docextractor/combine.go:35”,“error”:"read data/20230221/teams/noteam/channels/eg9ci9bda7r1pjgmjnre83tcxo/users/e7t9j5widtd4bqz7y5mq41c45c/4hbkgs5s3ffkmpe6a859ewdrby/Приказ о назн МРГ.pdf: file already close>
{“timestamp”:“2023-06-08 20:13:24.756 +05:00”,“level”:“warn”,“msg”:“unable to extract file content”,“caller”:“docextractor/combine.go:35”,“error”:“exec: "pdftotext": executable file not found in $PATH”}
{“timestamp”:“2023-06-08 20:13:24.756 +05:00”,“level”:“warn”,“msg”:“unable to extract file content”,“caller”:“docextractor/combine.go:35”,“error”:“error copying data into temporary file: write /tmp/pdflib1161241534: copy_file_range: use of closed file”}
{“timestamp”:“2023-06-08 20:13:24.756 +05:00”,“level”:“warn”,“msg”:“unable to extract file content”,“caller”:“docextractor/combine.go:35”,“error”:"read data/20230228/teams/noteam/channels/ojh5t14uzjr6mryrot4595r8re/users/e7t9j5widtd4bqz7y5mq41c45c/rmofe8psnp8c3bgfduaikrw3xe/Сметная подсистема в EnSpace.pdf: file al>
{“timestamp”:“2023-06-08 20:17:21.425 +05:00”,“level”:“info”,“msg”:“Post.Message has size restrictions”,“caller”:“sqlstore/post_store.go:2544”,“max_characters”:16383,“max_bytes”:65535}
OK, so there are several things that need to be taken care of.
{“timestamp”:“2023-06-08 20:13:13.655 +05:00”,“level”:“error”,“msg”:“Failed to install prepackaged plugin”,“caller”:“app/plugin.go:964”,“path”:”/opt/mattermost/prepackaged_plugins/mattermost-plugin-apps-v1.2.1-linux-amd64.tar.gz”,“error”:“Failed to install extracted prepackaged plugin /opt/mattermost/prepackaged_pl>
This might indicate write permission problems on the /opt/mattermost path, please run the following command to fix that:
chown -R mattermost: /opt/mattermost
{“timestamp”:“2023-06-08 20:13:24.005 +05:00”,“level”:“error”,“msg”:“Unable to activate plugin”,“caller”:“app/plugin.go:171”,“plugin_id”:“com.mattermost.apps”,“error”:“failed to load initial configuration: plugin requires Mattermost Site URL to be set”}
Your server’s SiteUrl configuration variable is not set. Depending on your deployment type, you need to either set it in the .env file or in your config.json or in the system console. It needs to point to the URL your clients will use to access the server (f.ex. https://mm.yourdomain.tld).
{“timestamp”:“2023-06-08 20:13:24.755 +05:00”,“level”:“warn”,“msg”:“unable to extract file content”,“caller”:“docextractor/combine.go:35”,“error”:“exec: "pdftotext": executable file not found in $PATH”}
The document extraction and indexing process requires some external tools, I’m not 100% sure, but you will need to at least install pdftotext and wv - others might be needed, but please install these two using your distribution’s package manager.
Thank you! Will try it now.
Should I restart something after all?
The import job died and is stuck in this state. The only way to clear it is by restarting the Mattermost server. Afterwards it might still show as “running”, but in fact, it isn’t, so just start a new import job then and watch the logs.
Hi again! Still have the same problem. Could you help please?
{“timestamp”:“2023-06-30 18:29:10.708 +05:00”,“level”:“info”,“msg”:“Skipping uploading of file because name already exists”,“caller”:“app/import_functions.go:1314”,“file_name”:“data/20230313/teams/noteam/channels/wbaahqozgi8y8qqyse1xqh161e/users/b4x5ckrz1f8tic7mjwb17xg8jw/6it798np7tg8zgu4578hmh8axe/Screenshot_2.png”}
{“timestamp”:“2023-06-30 18:29:10.753 +05:00”,“level”:“error”,“msg”:“Error attaching files to post.”,“caller”:“app/import_functions.go:1641”,“post_id”:“i36hkw69tpgijbufwjomadoi7o”,“post_file_ids”:[“rka547fcb3dejdkpx1cxp8pi5e”],“error”:“invalid input: entity: FileInfo field: <id, postId, creatorId> value: <rka547fcb3dejdkpx1cxp8pi5e, i36hkw69tpgijbufwjomadoi7o, zgu8567dft8jdcpycpk1r3g7fy>”}
{“timestamp”:“2023-06-30 18:29:10.852 +05:00”,“level”:“error”,“msg”:“Error attaching files to post.”,“caller”:“app/import_functions.go:1641”,“post_id”:“qsr5sci8jj869c3h4he8itcaiw”,“post_file_ids”:[“p44wornb9if1xn99fmcha7tcue”],“error”:“invalid input: entity: FileInfo field: <id, postId, creatorId> value: <p44wornb9if1xn99fmcha7tcue, qsr5sci8jj869c3h4he8itcaiw, aoyjcxbkupg9tkkaneninehifh>”}
{“timestamp”:“2023-06-30 18:29:10.854 +05:00”,“level”:“info”,“msg”:“Preparing file upload from ZIP”,“caller”:“app/import_functions.go:1270”,“file_name”:“data/20230425/teams/noteam/channels/babwzgpcnffjzd76uc6ze4btpr/users/zpmsn5rbwbb57r6aq4kokw4yth/so3qkkko4igc9nocekzuobkkie/image.png”,“file_size”:77634}
{“timestamp”:“2023-06-30 18:29:10.909 +05:00”,“level”:“info”,“msg”:“Skipping uploading of file because name already exists”,“caller”:“app/import_functions.go:1314”,“file_name”:“data/20230425/teams/noteam/channels/babwzgpcnffjzd76uc6ze4btpr/users/zpmsn5rbwbb57r6aq4kokw4yth/so3qkkko4igc9nocekzuobkkie/image.png”}
{“timestamp”:“2023-06-30 18:29:10.910 +05:00”,“level”:“info”,“msg”:“Preparing file upload from ZIP”,“caller”:“app/import_functions.go:1270”,“file_name”:“data/20230425/teams/noteam/channels/babwzgpcnffjzd76uc6ze4btpr/users/zpmsn5rbwbb57r6aq4kokw4yth/iwet5bgfm3fm9f7y9wpyye5kry/image.png”,“file_size”:101000}
{“timestamp”:“2023-06-30 18:29:10.975 +05:00”,“level”:“info”,“msg”:“Skipping uploading of file because name already exists”,“caller”:“app/import_functions.go:1314”,“file_name”:“data/20230425/teams/noteam/channels/babwzgpcnffjzd76uc6ze4btpr/users/zpmsn5rbwbb57r6aq4kokw4yth/iwet5bgfm3fm9f7y9wpyye5kry/image.png”}
{“timestamp”:“2023-06-30 18:29:11.009 +05:00”,“level”:“error”,“msg”:“Error attaching files to post.”,“caller”:“app/import_functions.go:1641”,“post_id”:“3itc3tndtfnijyegxzm4hhzj4w”,“post_file_ids”:[“e4q6wr8puid65bc9mhpc7u9oir”],“error”:“invalid input: entity: FileInfo field: <id, postId, creatorId> value: <e4q6wr8puid65bc9mhpc7u9oir, 3itc3tndtfnijyegxzm4hhzj4w, fhn1c3a4pt8rpyboqnf654by6o>”}
{“timestamp”:“2023-06-30 18:29:11.009 +05:00”,“level”:“error”,“msg”:“Error attaching files to post.”,“caller”:“app/import_functions.go:1641”,“post_id”:“5ujpmhgrnbfyxxzn75bfgxzrwa”,“post_file_ids”:[“pwyrokx5gff4jxg4eebbc5qfbh”],“error”:“invalid input: entity: FileInfo field: <id, postId, creatorId> value: <pwyrokx5gff4jxg4eebbc5qfbh, 5ujpmhgrnbfyxxzn75bfgxzrwa, fhn1c3a4pt8rpyboqnf654by6o>”}
{“timestamp”:“2023-06-30 18:29:11.013 +05:00”,“level”:“info”,“msg”:“Preparing file upload from ZIP”,“caller”:“app/import_functions.go:1270”,“file_name”:“data/20230414/teams/noteam/channels/e7nzbrwfetgz7qzrb9j5x6imxc/users/odjhau95fjro5r9xbigo5t9now/nr947egqijr8mg1s4h9z3ixkth/image.png”,“file_size”:121337}
{“timestamp”:“2023-06-30 18:29:11.085 +05:00”,“level”:“info”,“msg”:“Skipping uploading of file because name already exists”,“caller”:“app/import_functions.go:1314”,“file_name”:“data/20230414/teams/noteam/channels/e7nzbrwfetgz7qzrb9j5x6imxc/users/odjhau95fjro5r9xbigo5t9now/nr947egqijr8mg1s4h9z3ixkth/image.png”}
{“timestamp”:“2023-06-30 18:29:11.131 +05:00”,“level”:“error”,“msg”:“Error attaching files to post.”,“caller”:“app/import_functions.go:1641”,“post_id”:“63nx41rqcjbffdh1yhnp8qfsmr”,“post_file_ids”:[“1fxbnxokf7rt8eup8yu6tnwnwh”],“error”:“invalid input: entity: FileInfo field: <id, postId, creatorId> value: <1fxbnxokf7rt8eup8yu6tnwnwh, 63nx41rqcjbffdh1yhnp8qfsmr, e7t9j5widtd4bqz7y5mq41c45c>”}
{“timestamp”:“2023-06-30 18:29:11.153 +05:00”,“level”:“error”,“msg”:“Error attaching files to post.”,“caller”:“app/import_functions.go:1641”,“post_id”:“femghys1ifrkbefhh3qe7fwh8o”,“post_file_ids”:[“d9nhy9fbdibuuczw5m95jbbrrh”],“error”:“invalid input: entity: FileInfo field: <id, postId, creatorId> value: <d9nhy9fbdibuuczw5m95jbbrrh, femghys1ifrkbefhh3qe7fwh8o, aoyjcxbkupg9tkkaneninehifh>”}
{“timestamp”:“2023-06-30 18:29:11.233 +05:00”,“level”:“info”,“msg”:“Bulk import worker finished”,“caller”:“app/import.go:121”,“bulk_import_worker_id”:“aso3efih6frnic14eddn4us9cy”,“processed_lines”:717}
{“timestamp”:“2023-06-30 18:29:11.260 +05:00”,“level”:“info”,“msg”:“SimpleWorker: Job is complete”,“caller”:“jobs/base_workers.go:96”,“worker”:“ImportProcess”,“job_id”:“r3rhqfo55fgzfcsub6jipd7dhy”}
{“timestamp”:“2023-06-30 18:31:11.533 +05:00”,“level”:“info”,“msg”:“SimpleWorker: Job is complete”,“caller”:“jobs/base_workers.go:96”,“worker”:“ExpiryNotify”,“job_id”:“c8ximc9k6p8o7d41jz6wub1oga”}
And a lot of such messages…
These messages suggest that you already ran an import and that the data is not able to be imported because of that. I assume you want to start over fresh on a local installation, right? If so, please install a fresh mattermost server, then make a snapshot/backup of the installation and re-run the import. Whenever the import fails, revert to the sanpshot after you fixed the errors and try again then.