Slack import: mmctl CLI causes instance abortion

Troubleshooting Command Line Interface
1

Hi everyone and thanks for this amazing tool!

I have been trying to migrate from Slack to self-hosted mattermost on Ubuntu 20.04 with Mysql server v8.0 and apache. A fresh mattermost installation is found in /var/www/mattermost

So far I have tried lot of things without success at all and I decided to retire, but this post gave me some clues and I re-opened.

Important 1 The Slack import file is quite small (14 MB), with some attachments. It has just 6 users and 3 channels.

In summary what I have done is:

  1. Export the Slack workspace following slack instructions, yielding the starting zip file (let’s call it startingFile.zip)

  2. Create the Slack bot and use slack advanced exporter following official guide:

wget https://github.com/grundleborg/slack-advanced-exporter/releases/download/v0.4.0/slack-advanced-exporter.linux-amd64.tar.gz
tar -xvzf slack-advanced-exporter.linux-amd64.tar.gz 
./slack-advanced-exporter --input-archive startingFile.zip --output-archive isabelBioinfoHub-with-emails.zip fetch-emails --api-token xoxb-25062577487410374518-xUHGPutC9oT494FGSRnts
./slack-advanced-exporter --input-archive isabelBioinfoHub-with-emails.zip --output-archive isabelBioinfoHub-with-emails-and-attachments.zip fetch-attachments
  1. Use the latest version of mmetl to transform the slack export into mattermost import:
git clone https://github.com/mattermost/mmetl
cd mmetl
go build #go is installed and in the path, to build the mmetl binary
cd .. # to go back to the folder where zipfiles are
./mmetl transform slack --team Neuromol --file isabelBionfoHub-export-with-attachments.zip --output isabelBioinfo-mattermost-import.jsonl
zip -r isabelBioinfoHub-mattermost-import.zip data isabelBioinfoHub-mattermost-import.jsonl

This creates a zip file with the .jsonl and the directory(ies) data/bulk-export-attachments

  1. I use the mmctl-beta version provided in this post to first validate the zip file to upload and import.
wget 'https://community.mattermost.com/files/fiinwbxwppg68eftofwd5fw3hw/public?h=APwtyR_jhvepkzH8b0j-REDcX4cwkLGKyM48d0Xehh8' -O mmctl-beta.zip
unzip mmctl-beta.zip
mv mmctl mmctl-beta # rename the binary so that it has no problems with original mmctl
cp mmctl-beta /var/www/mattermost/bin
  1. Finally, I try to import the data, first validating the zip, then uploading and finally processing it using mmctl-beta (I have also tried with original mmctl). Note that the team Neuromol is created using the browser:
cd /var/www/mattermost
bin/mmctl auth login #to log in, using local-server and the system admin user created
bin/mmctl-beta import validate ~/isabelBioinfoHub-mattermost-import.zip --team Neuromol
# it yields no errors, see screenshot
bin/mmctl-beta import upload ~/isabelBioinfoHub-mattermost-import.zip #all ok
bin/mmctl-beta import process 5q97wa3ecb8mxfdie96zhr4gta_isabelBioinfoHub-mattermost-import.zip

imagen
imagen735×341 69.2 KB

I have the mattermost server session in an other terminal to debug, I have started the service like this:

sudo -u mattermost bin/mattermost # in port 8064 in this instance, with an apache proxy

And the instance get aborted, stopping and when I re-start the instance, the import process is stucked. Here I post a log of the mattermost instance:

imagen
imagen1106×146 39.4 KB

Then from another terminal, I run
bin/mmctl-beta import process 5q97wa3ecb8mxfdie96zhr4gta_isabelBioinfoHub-mattermost-import.zip # import log successfully created

And I link the long log here

Thank you very much in advance and sorry for bothering, but I have not been able to figure out myself.

2

I forgot to add that I could import the users, messages and channels by using mattermost CLI with the startingFile.zip (the slack original export).

3

Welcome to the Mattermost Forums @AdriTara

Using the mattermost CLI for imports is not recommended and is not actively maintained. mmctl import should be used (as you tried).

From the logs it seems like the team ID could not be found and for some reason the server tried to import the user anyway. Does the team “Neuromol” exist on the instance you are trying to import into?

4

Thanks for having a look and your response!

I know that mattermost CLI is not recommended nor maintained, but I tried and worked so at the moment is what I have done so far.

Regarding your question, yes, the team is created using the browser. Indeed, some users are imported, but the import status remains “in_process” forever.

5

Hi,
I have been trying but still no luck. I have no answer about this, I was wondering if @plusmid had any chance to have a deeper look and give me any suggestion to try.

6

I am experiencing the same exact issue I think. Extract from the console log:

{"timestamp":"2022-10-20 11:19:49.856 +02:00","level":"debug","msg":"Worker received a new candidate job.","caller":"jobs/base_workers.go:50","worker":"ImportProcess"}                                                                            panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5581842c69f9]

goroutine 7414 [running]:
panic({0x558184f75660, 0x55818631ab30})
  runtime/panic.go:987 +0x3ba fp=0xc0044a2880 sp=0xc0044a27c0 pc=0x558182c75b3a
runtime.panicmem(...)
  runtime/panic.go:260
runtime.sigpanic()                                                                                                                                                                                                                                         
  runtime/signal_unix.go:835 +0x2f6 fp=0xc0044a28d0 sp=0xc0044a2880 pc=0x558182c8ca76
github.com/mattermost/mattermost-server/v6/app.(*App).importUserTeams(0xc0028fd948, {0x55818526bac0, 0xc0027ef0e0}, 0xc002b41800, 0xc0012d02e8)

The import manages to create the channels, and then gets stuck after having imported 2 users. I am suspecting that it has something to do with the Team name. The Mattermost database has 2 names for teams in the Teams table, one is the “DisplayName” and the other is the “Name” which is used in the URL. In my case the “DisplayName” is the “Name” except that the first character is uppercase. Inside the JSON that I am trying to import the upper case version of the name is reference. Also weird is that it manages to import the channels, which also reference the same team name, and it only breaks once the second user is imported. Weirdly, the 2 users that are imported don’t get assigned to the team that they should, however, the channels do get assigned.

I have tried recreating the team that I am importing into, but that does not seem to change anything.

7

Something strange is going on here. The server requests a list of teams from the database. The DB returns a list of teams matching the request. The server tries find the team in the list but can’t find it. There seems to be a mismatch somewhere.

8

So, my other response was hidden by the spam filter, but I just figured on the problem in my case thanks to the hints above. When I transformed the backup using mmetl I set the name to “Team” instead of “team” (capitalized), which matches the DisplayName property instead of the Name (URL) property. It’s probably the same problem here.

9

Hey @zenhaeus, welcome to the forums.
I think you are onto something here. Depending on the default collation, the database might match the team name case insensitive. The server itself always matches case sensitive.

@AdriTara could you check the default collation for your database, please?

10

Have you re-converting the export with mmetl with the lower case name? Or did you end up changing the team on the server. Which collation do you use in your database?

11

Yes, that’s exactly what I did, sorry, maybe I wasn’t very clear in my response above, but reconverting the export with lower case name fixed it.

I use the utf8mb4_general_ci collation in a MariaDB database. I have noted however that some individual tables were set to utf8mb4_unicode_ci, which resulted in some warning entries in the server log that stated that there was a collation mismatch. After setting all tables to utf8mb4_general_ci these warning disappeared.

12

Okay, thanks for confirming this. The server code assumes that the match is case sensitive while we allow the database to be configured to be insensitive. That leads to the result not being matched properly and the import crashes. I’ll write a fix for this.

Until the fix is released, make sure the casing matches the team name (not display name!) to prevent this error.

Thanks to @AdriTara and @zenhaeus for bringing this up and helping with solving the mystery.

13

Hey all… Just wanted to let you know I was facing exactly the same issue on the docker image preview. After changing the team name to lowercase everything went through.

Thanks all!