Support for team import/export

Hi everyone!

I’d like to get your thoughts on a design proposal to enable admins to migrate a single team from one instance to another. I’m aware of various communities where the lack of this feature is a significant problem and I would be interested to work on it. Because it’s a significant piece of work with various design questions, I think it’s worth coordinating before, so I don’t spend too much time implementing something that has no chances of getting merged.

Problem statement

As a team admin in Mattermost, I want to be able to download an archive containing the data that makes up my team and import it in another Mattermost instance (concerning private data, see the “Design details” section below).

Broader context

Currently, Mattermost only lets system admins generate a dump of the entire instance (containing all teams), via the mmctl export command. There is no built-in way to filter the dump to only contain a single team. As a result, hosting providers which host multiple teams in the same instance (such as framateam.org or girofle.cloud) are unable to offer migration options for their users. This goes against the portability expectations associated with an open source platform and might also be incompatible with regulatory requirements in some jurisdictions or industries.

Relevant UserVoice feature requests

Team admin should be able to export datas

Mattermost Bulk Export Single / Selected Team

High-level design

Unlike the current instance-wide export functionality, team exports should ideally be available to team admins via the web UI. In the “Team Export” dialog, a new “Import / Export” tab is added. In this tab, the team admin is able to:

  • Generate a new export. This triggers a job to prepares the corresponding archive. For a given team, there can only be one export job at a time. The archive is made available for download once the job is complete. Only the archive generated by the latest export job can be downloaded and the time at which it was generated is shown. It expires after some delay.
  • Upload an archive for import in the current team. This also starts a background job to perform the import. A team can only have one import job at a time and the dialog shows the state of such a job, if any.

Design details

User management

One major hurdle consists in the fact that user accounts are defined at the instance-level and not team-level. Despite that, users need to be exported and imported as part of a team export.

When exporting a team, user accounts belonging to this team are exported as well. Users who have left the team should likely not be exported, meaning that their interactions will then appear as deleted users in the imported team (to rejoin the team after migration, they need to create a new account).

When importing a team, one needs to merge the list of users supplied in the archive with the list of users already present in the Mattermost instance. The usernames supplied in the archive need to be changed in two cases:

  1. An existing user has the same email address but a different username. In this case, the username of the existing user is used instead of the one in the archive
  2. An existing user has the same username but a different email address. In this case, we generate a new username for the user to be imported (appending a number to the username and incrementing this number until the username is free).

Once new usernames have been assigned to the users to be imported, we need to rewrite the rest of the archive’s contents to update them with the new usernames where needed. A logic similar to the one already implemented in mmetl (for Slack import) can be used.

One could consider giving the opportunity to the user to provide a mapping from old to new usernames themselves but I think this is rather complex so I would leave this out of a first version of this feature.

Handling of private data

It is an important product decision to determine whether private channels and direct messages should be part of a team export, as including them makes them accessible to the team admin.

Similarly, the export will expose personal information from users that team admins might otherwise not be able to see, for instance the email address or full name if the “Show Email Address” or “Show Full Name” options have been turned off.

To address those issues, the generation of exports could be guarded by a permission flag (or multiple flags) that could be configured on the team admin role by the system admin. Or make team export and import only accessible to site admins (which does not introduce any privacy issue since they already have access to this data through the existing instance-wide export).

Archive format

The export format for a team should ideally follow that of an instance-wide export. The ability to use a team export as an instance export containing a single team would simplify the user experience, avoiding to create multiple types of exports which need to be imported in different ways.

If a team export is indeed the same thing as a single-team instance export, then care needs to be taken that importing such an export by a team admin only accepts dumps that contain a single team and is not able to make changes to any other teams than the current one.

User opt-in

It could be useful to let users opt in or out of their messages being included in a team export, especially if the export feature is available to team admins (and not just system admins). This is makes the export process rather complicated but likely better compliant with privacy regulations such as GDPR. In that case, more work would be needed to design how user consent should be gathered (for instance via custom user metadata field or a bot similar to feedbackbot). Also, one would need to decide how to represent the messages of users who opted out after the migration.

Implementation strategy

The implementation could be broken down into the following tasks:

  1. Add a --team option to the mmctl export create command, implementing the corresponding filtering
  2. Add support for automatically renaming users during an import to avoid integrity violations as outlined above. This feature would also be available when importing entire instances with multiple teams.
  3. Expose team export from the web UI, possibly introducing the necessary permission to control availability of the feature and the opt-in mechanism for users.
  4. Expose team import from the web UI.

I would intuitively implement this in Mattermost directly, but some or all the parts could potentially be developed as a plugin, or an external tool similar to mmetl.

Looking forward to your feedback on all that!

1 Like

Thanks for the proposal @wetneb! I think being able to export selected teams is a great idea and would love to see it happens. Per the GDPR related feature, that’s something needs more discussion and thoughts. Do you think these two things can be separately implemented?

We can discuss the details on how users would get imported or how we are going to approach DMs/GMs once we resolve the question above.

Looking forward to hearing from you.

1 Like

Thanks for your interest!

By “GDPR feature” do you mean the ability for users to opt in or out of a team export?
To be clear, I don’t know if that would actually be required by the GDPR - I am far from an expert on this topic.

As mentioned in the implementation plan, I would intuitively start by making this functionality available to system admins (with steps 1. and 2.), which would just be team-restricted version of the existing instance export, so that wouldn’t depend on any new opt in/out mechanism.

1 Like

Yes, the first point was about opt-out specific users. Let’s phase that out for now, I think we should discuss the requirements and expectations separately. This can be a very complicated feature anyway.

As for the implementation plan, there are several things that needs to be addressed. First of all do you want to include DMs/GMs into this? Another thing is, this may require refactoring many export methods within the code (application and database layers) to take teamId as an option. The queries will change depending on the data type (eg. teamId for channels and teamMembership for DMs and so on).

Another thing is that exporting roles, schemes, emojis etc which are not tied to teams. It wouldn’t harm to import those over and over again but then the risk of overwriting should be clarified.

I think this should be sysadmin level privilege because normally team admins doesn’t have system level information (eg. accessing the DB) and being able to access DMs by team admins is not something normally achievable.

Looking forward to hearing your thoughts and plans on these.

Most people I got feedback from seem to agree on the idea that DMs/GMs shouldn’t be exported. Intuitively, when migrating a team, you’re primarily trying to preserve the group’s accumulated knowledge, and DMs/GMs aren’t really contributing to that in my experience. Plus they can be private and sensitive of course. I would still include private channels though. Does that sound sensible?

Interesting, I didn’t think about those yet. Here are my intuitions:

  • schemes: in the OSS version there is only a system scheme, which I would leave out of the team export. The migrated team would then inherit the settings from the system scheme in place in the new instance.
  • roles: I haven’t been able to find out how new roles can be created by users. From what I can see, in the OSS version they are entirely predefined, so I would leave them out of the export.
  • emojis: I would export all custom emojis of the instance, and accept the risk that they can overwrite emojis in the importing instance. Or do you think we need to introduce a deduplication logic similar to the one needed for users?

I think that makes sense, even if we leave DMs out of the export. It’s something that could be reconsidered in a next iteration if need be, but I think the feature would already be very useful even if restricted to site admins.