URGENT: Old directly uploaded images missing after setting server to use S3

Troubleshooting
1

We’re running Mattermost 5.13.2 and had previously allowed direct uploads to the hosting server to the ./data/ folder. A few days ago we changed the File Storage method to Amazon S3. Now we’re having a consistency problem where images that were uploaded previous to this change are missing or only showing small preview images.

I’m guessing that Mattermost somehow does not handle old uploaded images properly due to some bug after changing to S3. Is there a way to address and fix this so that old direct upload content AND new S3 upload content are BOTH accessible?

Screenshot of a message with missing image:

And multiple images:

2

@ylluminate, what is this setting set to on your server https://docs.mattermost.com/administration/config-settings.html#allow-untrusted-internal-connections-to and do you see errors in the logs?

3

config/config.json has:

"EnableInsecureOutgoingConnections": false,
"AllowedUntrustedInternalConnections": "",

And I see no output within logs/mattermost.log or logs/notifications.log…

4

@ylluminate, the server doesn’t support using two file stores at the same time: you’ll ned to copy all the data from the old ./data folder into S3 (using the same directory structure).

1 Like
5

@jesse unfortunately I don’t quite understand how that would work in terms of the expected directory structure: Are you saying that I can use an S3 client and just drop the data/* folders (not data itself, but its children) into the bucket and voila, it should work sans hitch?

EDIT: So from what I can tell after reviewing folder structures this is what you mean @jesse so I’m going to give it a shot here since we’re in an urgent situation. :crossed_fingers:

UPDATE + SOLUTION:

So yes, I was able to use an S3 client and upload all of the locally stored ./data/* folders and all data is accessible again.

@amy.blais I would highly, highly suggest that someone add this bit of instruction at the top of the /admin_console/environment/file_storage page so that it’s very clear as to what will happen once the file store is changed. It is unfortunate that Mattermost doesn’t support both local and S3 stores simultaneously, it should be relatively straightforward to add, but clearly this has just been an oversight so far and this kind of danger needs a nice solid warning since the resultant behavior was not at all expected during the config change…

6

Glad it worked, @ylluminate, and I’m sorry for the trouble it caused during the transition. I’ve filed https://mattermost.atlassian.net/browse/MM-18129 to get our team to investigate your exact proposal.

7

Hey that’s awesome @jesse! Thanks! It was a stressful situation for a little bit here with some folks griping about missing everything mid-project meeting… :smiley: Just never expected this.