Bulk import with existing users

Troubleshooting Command Line Interface
1

Hi all,

The latest posts on this issue are a few years old now, so I create a new one.

Can the bulk import tool deal with users that already exist in mattermost? For me it fails with
“app/user.go:317”,“Couldn’t save the user”,“error”:“SqlUserStore.Save: An account with that email already exists.”

I have also tried the command line import slack that can merge the users, but it only imports private messages to users that have not yet been messaged on Mattermost. Existing chats are skipped.

I’m on Mattermost 5.23.0 Team edition with postgresql 9.4 as database.

Thanks in advance

2

Hello, @bartels

Based on a discussion with one of the engineering team members, the import tool should be able to handle existing users by updating them with whatever changed.

However, I would like to further understand this statement:

  • Were you originally referring to the bulk loading tool? If yes, what was the outcome of running the following command prior to running the bulk load command?
sudo -u mattermost bin/mattermost import bulk data.jsonl --validate

  • Can you please confirm what do you mean by command line import slack that can merge the users? Which documentation / article that you were referring to when you tried this?

  • Were you trying to upload the .jsonl from another communication tool into Mattermost (Slack)? If yes, which method did you try based on the Migrating from Slack documentation?

3

Hello @ahmaddanial,

Thanks for your reply. Sorry if things were unclear, I’ll provide more detail.

We are running a workspace that was migrated from Slack with the Standard export that only contains users and public channels. We have been using that one for a while but now got our Slack Corporate export and also want to import our Slack direct message history. That means that all the users already exist and most of us already have some direct messages between us in Mattermost. Now we want to merge the messages from Slack.

I was following the instructions under
https://docs.mattermost.com/administration/migrating.html#migrating-from-slack

I have tried 2 approaches:

  1. Bulk import tool

I ran the slack-advanced-exporter to include the attachments into the exportet .zip file (Only the command to add attachments, not mail addresses. These are already present in the export and the linked necessary API Tokens seem to be deprecated.). I then used the mmetl tool to turn the .zip file with attachments into a .jsonl which finishes successfully.

Running
sudo -u mattermost bin/mattermost import bulk workspace.jsonl --validate
shows that everything is OK:
Validation complete. You can now perform the import by rerunning this command with the --apply flag.

But if I run with --apply it fails:

Running Bulk Import. This may take a long time.

{"level":"error","ts":1590402795.3863447,"caller":"app/user.go:279","msg":"Couldn't save the user","error":"SqlUserStore.Save: An account with that email already exists., user_id=fr4w5m1ijb8g5gomdijyep7p5e, pq: duplicate key value violates unique constraint \"users_email_key\""}
SqlUserStore.Save: An account with that email already exists., user_id=fr4w5m1ijb8g5gomdijyep7p5e, pq: duplicate key value violates unique constraint "users_email_key"
Error occurred on data file line 14
{"level":"info","ts":1590402795.5344722,"caller":"app/server.go:402","msg":"Stopping Server..."}
{"level":"info","ts":1590402795.534502,"caller":"app/web_hub.go:130","msg":"stopping websocket hub connections"}
{"level":"warn","ts":1590402795.5345318,"caller":"app/web_hub.go:135","msg":"We appear to have already sent the stop checking for deadlocks command"}
{"level":"info","ts":1590402795.534698,"caller":"app/plugin.go:279","msg":"Shutting down plugins"}
{"level":"info","ts":1590402795.535528,"caller":"app/server.go:453","msg":"Server stopped"}
Error: SqlUserStore.Save: An account with that email already exists., user_id=fr4w5m1ijb8g5gomdijyep7p5e, pq: duplicate key value violates unique constraint "users_email_key"

Everytime I run the command a different user_id shows the violation.

  1. Migrating from Slack using the Mattermost CLI

This one does not seem to avoid duplicates, so I removed the public channels from the exported zip which have already been migrated to Mattermost.

I then ran
sudo -u mattermost bin/mattermost import slack myTeam slackExportWithAttachments.zip

That produces outputs like:

Slack user merged with an existing Mattermost user with matching email
Unable to import Slack channel .
Unable to import Slack channel mpdm-XXX(direct messages with multiple participants)
Notes:
=======

- Some messages may not have been imported because they were not supported by this importer.
- Slack bot messages are currently not supported.
- Additional errors may be found in the server logs.

Finished Slack Import.

This manages to import direct messages even with attachments, but only in those chats where no messages have been sent on Mattermost yet. If messages between the users already exist, no messages from Slack are merged into the chat.

I hope, this provides sufficient detail.

Thank you in advance!

4

Hi, @bartels

No worries. I engaged the team to discuss about this matter and according to @sven.huester , the error pops up if the user in question has the same email but different username. So, both fields need to be the same to allow overwriting of the account.

Can we confirm on this before we attempt to run the import again?

In regards to this statement, does that mean that you are overwriting an existing Slack-imported instance with the latest information (users + public channels + DMs):

Just trying to understand the scenario better since I believe that the bulk import is not a synchronization tool based on this statement in the documentation.

Additionally, can you provide more information of the user fr4w5m1ijb8g5gomdijyep7p5e from both the Slack export and Mattermost database for comparison?

For Mattermost, run the SQL query below on the database and share the information (with sensitive data removed such as password):

SELECT * FROM Users WHERE Id = "fr4w5m1ijb8g5gomdijyep7p5e";
5

Thank you both @ahmaddanial and @sven.huester,

I did not know that the usernames had to be identical in the import. I think at least one user changed the name, so I could manually change that in the imported jsonl file.

Users and public channels have already been imported from our first Slack export. I was hoping to now also import the DMs and private channels from the Slack corporate export. Ideally without removing any channels and DMs that have been created in Mattermost in the meantime.

I do not really understand the synchronization statement about the bulk import. I’m not trying to remove information, just add messages and channels.

As I said in my last post, the Id of the violating user changes every time the bulk import tool is executed. I think it is the Id of a new user it tries to create. The given Id appears neither in our mattermost SQL database nor the .jsonl import file.

6

The bulk import never removes data. Only writes new data. Existing channels will be overwritten with identical data and differing data will be added.
You can for example delete the public channels from the import file and only import the private ones if that is what you’re going after.

Hope this clears it up a little.

Best,
Sven

1 Like
7

I have now manually edited the changed usernames in the jsonl file. Indeed, the bulk import succeeds and seems to properly merge the direct messages. Thank you very much!

The only thing surprising me is that it seems to overwrite user passwords although none are present in the jsonl file. Can that be disabled? If not I can still ask my users to have their passwords reset after the import.