Moving Files from an existing disk store to s3

Troubleshooting
1

Hello,

I have a mattermost team instance that is configured to use the default file system location to store attachments. The server has been in use, and there are files in the data directory? How would I move the existing files to S3?

2

Would also be interested in a solution.

3

Hi @xingliu and @yoshtec,

As long as you keep the folder structure of the data folder the same as the structure of the S3 bucket (for example, data/teams/12345 becomes teams/12345 in the S3 bucket), you should be able to just switch your Mattermost instance to use S3 for file storage without problems. If you use the AWS Command Line Interface, there’s an aws s3 sync command that should be able to copy everything over for you

1 Like
4

It really is this simple! I quickly set up Mattermost with local storage to escape from Slack. It’s been wonderful! I wanted to start uploading more files but I was concerned about space. S3 is the obvious solution. I migrated painlessly.

Here are some notes from my migration…
AWS IAM policy:
[from AWS S3: The right policy to set - Antoine Aflalo]

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::BUCKET_NAME"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::BUCKET_NAME/*"
            ]
        }
    ]
}

$ aws s3 sync /opt/mattermost/data/ s3://BUCKET_NAME

System Console → Environment → File Storage
After configuring for S3, you have to save the configuration before testing it.

Changes seemed to take effect immediately after testing. (I uploaded a photo and it saved in S3.) I did restart the server to ensure it was still working properly, though,

systemctl restart mattermost

Thanks, All!

1 Like
5

Thank you for the steps.
Notes:

  1. If you are not doing the above steps as a mattermost user I suggest executing following command,
    sudo usermod -G mattermost <current-user>
    sudo chmod -R 750 /var/opt/mattermost/data
  2. And you will have to restart services to take effect the changes.
6

To move the existing files from the default file system location to Amazon S3 in your Mattermost team instance, you have a few options:

1- Using an S3 migration tool : There are third-party tools specifically designed for migrating files from a local file system to Amazon S3. These tools often provide a user-friendly interface and automate the migration process. One such tool is “s3-syncer,” which you can find on GitHub. also you can try Gs Richcopy 360 or ShareGate .

2- Using the AWS Command Line Interface (CLI): The AWS CLI provides a comprehensive set of commands to interact with various AWS services, including S3. You can use the aws s3 sync command to synchronize the files from your local directory to an S3 bucket.

3- Manual upload using S3 tools: If you prefer a more manual approach, you can use S3 tools like s3cmd or the AWS Management Console to upload the files manually. With s3cmd , you can use the put command to upload files to your S3 bucket.