Hi everyone!
I’d like to get your thoughts on a design proposal to enable admins to migrate a single team from one instance to another. I’m aware of various communities where the lack of this feature is a significant problem and I would be interested to work on it. Because it’s a significant piece of work with various design questions, I think it’s worth coordinating before, so I don’t spend too much time implementing something that has no chances of getting merged.
Problem statement
As a team admin in Mattermost, I want to be able to download an archive containing the data that makes up my team and import it in another Mattermost instance (concerning private data, see the “Design details” section below).
Broader context
Currently, Mattermost only lets system admins generate a dump of the entire instance (containing all teams), via the mmctl export
command. There is no built-in way to filter the dump to only contain a single team. As a result, hosting providers which host multiple teams in the same instance (such as framateam.org or girofle.cloud) are unable to offer migration options for their users. This goes against the portability expectations associated with an open source platform and might also be incompatible with regulatory requirements in some jurisdictions or industries.
Relevant UserVoice feature requests
Team admin should be able to export datas
Mattermost Bulk Export Single / Selected Team
High-level design
Unlike the current instance-wide export functionality, team exports should ideally be available to team admins via the web UI. In the “Team Export” dialog, a new “Import / Export” tab is added. In this tab, the team admin is able to:
- Generate a new export. This triggers a job to prepares the corresponding archive. For a given team, there can only be one export job at a time. The archive is made available for download once the job is complete. Only the archive generated by the latest export job can be downloaded and the time at which it was generated is shown. It expires after some delay.
- Upload an archive for import in the current team. This also starts a background job to perform the import. A team can only have one import job at a time and the dialog shows the state of such a job, if any.
Design details
User management
One major hurdle consists in the fact that user accounts are defined at the instance-level and not team-level. Despite that, users need to be exported and imported as part of a team export.
When exporting a team, user accounts belonging to this team are exported as well. Users who have left the team should likely not be exported, meaning that their interactions will then appear as deleted users in the imported team (to rejoin the team after migration, they need to create a new account).
When importing a team, one needs to merge the list of users supplied in the archive with the list of users already present in the Mattermost instance. The usernames supplied in the archive need to be changed in two cases:
- An existing user has the same email address but a different username. In this case, the username of the existing user is used instead of the one in the archive
- An existing user has the same username but a different email address. In this case, we generate a new username for the user to be imported (appending a number to the username and incrementing this number until the username is free).
Once new usernames have been assigned to the users to be imported, we need to rewrite the rest of the archive’s contents to update them with the new usernames where needed. A logic similar to the one already implemented in mmetl
(for Slack import) can be used.
One could consider giving the opportunity to the user to provide a mapping from old to new usernames themselves but I think this is rather complex so I would leave this out of a first version of this feature.
Handling of private data
It is an important product decision to determine whether private channels and direct messages should be part of a team export, as including them makes them accessible to the team admin.
Similarly, the export will expose personal information from users that team admins might otherwise not be able to see, for instance the email address or full name if the “Show Email Address” or “Show Full Name” options have been turned off.
To address those issues, the generation of exports could be guarded by a permission flag (or multiple flags) that could be configured on the team admin role by the system admin. Or make team export and import only accessible to site admins (which does not introduce any privacy issue since they already have access to this data through the existing instance-wide export).
Archive format
The export format for a team should ideally follow that of an instance-wide export. The ability to use a team export as an instance export containing a single team would simplify the user experience, avoiding to create multiple types of exports which need to be imported in different ways.
If a team export is indeed the same thing as a single-team instance export, then care needs to be taken that importing such an export by a team admin only accepts dumps that contain a single team and is not able to make changes to any other teams than the current one.
User opt-in
It could be useful to let users opt in or out of their messages being included in a team export, especially if the export feature is available to team admins (and not just system admins). This is makes the export process rather complicated but likely better compliant with privacy regulations such as GDPR. In that case, more work would be needed to design how user consent should be gathered (for instance via custom user metadata field or a bot similar to feedbackbot). Also, one would need to decide how to represent the messages of users who opted out after the migration.
Implementation strategy
The implementation could be broken down into the following tasks:
- Add a
--team
option to themmctl export create
command, implementing the corresponding filtering - Add support for automatically renaming users during an import to avoid integrity violations as outlined above. This feature would also be available when importing entire instances with multiple teams.
- Expose team export from the web UI, possibly introducing the necessary permission to control availability of the feature and the opt-in mechanism for users.
- Expose team import from the web UI.
I would intuitively implement this in Mattermost directly, but some or all the parts could potentially be developed as a plugin, or an external tool similar to mmetl
.
Looking forward to your feedback on all that!