S3 error after upgrade to 3.5.0

Summary

After upgrading from 3.4.0 to 3.5.0, I am unable to add new profile pics.

Steps to reproduce

  • Upgrade from 3.4.0 to 3.5.0.
  • Upload user profile picture

Expected behavior

Profile picture successfully upload

Observed behavior

Original s3 endpoint set to “s3.amazonaws.com” in 3.4.0 and integration worked fine.

User receives “Could not upload profile image” error message
Server logs show:

[details: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.]

When changing the storage endpoint to the aws listed endpoint xxx.s3-website-us-west-2.amazonaws.com (other variants tried), the error changes to:

[details: Amazon S3 endpoint should be ‘s3.amazonaws.com’.]

Config file appears correct (bucket redacted):
“AmazonS3Bucket”: “xxx”,
“AmazonS3Region”: “us-west-2”,
“AmazonS3Endpoint”: “s3.amazonaws.com”,
“AmazonS3SSL”: true

I made some progress on this. Reading through the code, it appeared that the minio-go package was checking for an endpoint located at “s3.amazonaws.com”. Looking at amazon’s endpoint documentation, I found that that had to be in region US Standard.

I sync’d my s3 bucket into a new bucket located in region US Standard and pointed mattermost to the new bucket. It’s not connecting properly, but none of the files are visible (they show up as :ok_hand:t3:).

It looks like there is a new database structure to store files, in a “fileinfo” table and that older format files should automatically convert, but it doesn’t seem to be happening. I don’t see any notes in the changelog. Does anyone have any experience with this problem?

Edit: newly added files appear correctly and add an entry into the fileinfo database

Hi @dwsc,

Sorry for the delayed response. Did you continue using 3.5 or did you roll back to 3.4 after that happened? And do you have a backup of your database from before the upgrade? Unfortunately since your file settings were misconfigured when you first ran 3.5, it removed the old references to the files from your posts and failed to add the new ones to the FileInfo table. Your files still exist on S3, but Mattermost doesn’t know about them any more.