Hi
It’s possible to be memory pressure although the x64 VM I am now running is the same 1gb and that never gets over 4-500 mb real… I have encountered occasional swap usage but tested by clearing that (swapoff -a / swapon -a) and then uploading an image. Still stuck and even with small images, the wait seems excessive (10-15 secs).
I guess I’m just curious regarding what processing is required other than thumbnail(s)? I don’t have any plugins enabled on the Pi.
Rgds
I don’t think there is any further processing beyond images and plugins.
Hello, please see here for the details I’ve collected as well, as I’ve run into this as well on a Raspberry Pi 4. My 4GB RAM is far from being full. No swap at all, I have it disabled.
This problem seems to be purely CPU-bound. Not RAM, and not disk-IO-bound. I’m watching what’s happening with both htop, and iotop. I tried migrating Mattermost’s data folder to a good-quality USB 3 stick, to split up the disk IO. That didn’t help at all.
I’m not running any additional Plugins beyond a basic install. No Clam-AV plugin.
I would sure appreciate a radio button somewhere in the System Console to make that image processing very lightweight, or not at all (as in, just attach the file, no matter what the file extension). Then let the web browser itself, or smartphone OS itself (with some Gallery-like app) show an image, using it’s own image-rendering ability (and the user must press the back button to go back to Mattermost).
I did my Mattermost install, using these (docker-avoiding) instructions. I’m on Raspbian Debian 10, with Mattermost 5.21, and MariaDB.
Here’s what htop looks like, during the image processing (and you can see the RAM is just fine, and the swap is off):
It seems the 398% is a summation of the next 4 lines below it…
BTW: Uploading a .jpg just smaller than 1MB required 46 seconds of maxed-out CPU time (just after upload to server, but just before user taps Enter to post), and nothing else running on the Raspberry Pi 4 was competing with Mattermost!
I strongly suspect that this crippling performance hit on a Raspberry Pi 4 has to do with how the NPM library “image-webpack-loader” gets used. It in turn calls “imagemin”. Imagemin performs the “wonderful” service of “Minify images seamlessly”.
It seems Mattermost is transparently compressing the images, before storing them! But formats like .jpg, .gif, and .png are already (internally) compressed. How does that make any sense? Spending all those CPU cycles, just to save like 1% off the file size? Please, say it ain’t so!
Please, can anyone explain how this transparent compression of images can be disabled?
Mattermost devs, could there please be a radio button for this, in the Mattermost System Console, or even a variable in config.json (which I don’t mind hand-editing)?
In image-webpack-loader (see link above), there is an option to “disable” it, but how to gracefully do this disabling, without hand-compiling all of Mattermost from source?
> In your webpack.config.js, add the image-loader, chained after the file-loader:
>
> rules: [{
> test: /\.(gif|png|jpe?g|svg)$/i,
> use: [
> 'file-loader',
> {
> loader: 'image-webpack-loader',
> options: {
> bypassOnDebug: true, // webpack@1.x
> disable: true, // webpack@2.x and newer
> },
> },
> ],
> }]
If you don’t believe me (that re-compressing .gifs, .jpgs, and .pngs is a bad idea), then please try compressing a few .gifs, .jpgs and .pngs laying around on your computer for yourself, and you’ll see that I’m not kidding.
.svg, OTOH, is not compressed by default. It makes sense to compress pretty much this image format only, as it just contains a bunch of XML text (which is very compressible).
The test above could intelligently call image-webpack-loader just for the .svgs, but then also disable image-webpack-loader, for the .jpgs, .pngs, and .gifs
Edit:
Yes indeed, in the source tree (v5.21.0), look at mattermost-webapp/webpack.config.js. Compression is turned on for those already-compressed file formats, see starting from line 196:
{
test: /\.(png|eot|tiff|svg|woff2|woff|ttf|gif|mp3|jpg)$/,
use: [
{
loader: 'file-loader',
options: {
name: 'files/[hash].[ext]',
},
},
{
loader: 'image-webpack-loader',
options: {},
},
],
},That all makes perfect sense @esbeeb - thank you for doing this and identifying a strong candidate!
Over to Devs to please, please provide what would appear to be a simple patch assuming this pans out…
I took a look at each of those formats.
Some are certainly compressed (png gif mp3 jpg eot woff2 woff), some are certainly not compressed (svg ttf), and one is annoyingly sometimes compressed (tiff).
I’m not sure whether it would be wise to one-sidedly compress all tiffs or not. My sensibility is that disk is cheaper than CPU (and people’s time waiting for compression unnecessarily, is the most expensive thing of all).
So I would personally incline towards just storing all tiffs, with the assumption that they are already internally compressed (and not compress them with image-webpack-loader). If a few uncompressed tiffs get through, then oh well, the disk is cheap enough. tiff is a pretty uncommon case these days, so I say it’s no biggie either way.
@justinegeffen, what do you think of this attachment-compressing issue, as stated above? I would argue that making instructions to install Mattermost on a Raspberry Pi are well and good, but sort of futile because the end users are likely to get seriously frustrated once the Raspberry Pi gets paralyzed on, say, a 2.8MB image upload.
@esbeeb if you still have your instance available, could you confirm that it is only impacting images for you? I had a similar issue uploading a 10mb SQL backup file. I would have expected no issue but the system locked solid.
That may have been an outlier and I didn’t re-attempt because I aborted the experiment and re-homed on an x64 VM. My preference is still to host on the Pi because that’s extremely efficient to run.
Thanks again
Mike
Hi @esbeeb,
Thanks for raising this, it’s really helpful feedback! @crspeller, do you have any suggestions around this? I found some some discussions around it on our Community server but nothing definitive.
Thanks!
Thanks for your comments, @justinegeffen and @b3lt3r.
I’ve created a user account in the Contributors Community channel myself, saying more there:
https://pre-release.mattermost.com/core/pl/h3sqfoaeriy4tmo5yohuocq7yo
image-webpack-loader and imagemin are only used for static images bundled with the web app (icons, emojis, logos, etc), and they’re only run at compile time. They aren’t used for file uploads, so they wouldn’t have any effect on them.
The processing that’s done on image uploads is to resize them so that they can be displayed in thumbnails in the apps without taking as much bandwidth. For anything other than small images, this is a significant bandwidth savings. That only applies to images though, so if you’re seeing performance issues with other file uploads, there’s something else causing issues here.
It turns out I was wrong as to which code is making the Raspberry Pi slow down so much. So my PR is certainly no fix. (removed the link above) Plus, it seems I was wrong to assume that Mattermost is compressing already-compressed image files when a user uploads an image.
The CPU-intensive code is shown here:
I’m grateful to jupenur and @hmhealey for tracking down the relevant image-processing code. What that code does, according to jupenur, is:
"… the explicit image processing we do in file.go on the server [not the webapp]. That would also still line up with the testing you did using .xcf files, because HandleImages is only invoked on files with an image/* MIME type – whereas the MIME type for .xcf is (on many systems) application/x-xcf .
Edit: In fact even more likely the root cause is in postprocessImage in file.go ."
Wild and crazy idea. Could the video hardware, which the RPi4 has, be used to do hardware-accelerated rendering of the images? Is there some easy way to say to the video hardware “please resize this image down to the following thumbnail sizes”?
Are there any video nerds out there?
There are go bindings for SDL.
There is a golang binding for FFMPEG.
The Raspberry Pi foundation is apparently also working on a hardware-accelerated FFMPEG library, for the Raspberry Pi.
That’d be interesting to investigate. I’ve always liked the idea of integrating some sort of video transcoding into the server as well, and that would almost certainly involve FFMPEG.
I’ve asked for a clue, here.
There is a tutorial on how to use the hardware-accelerated SDL library on the Raspberry Pi.
I re-tried my install instructions for both Ubuntu 18.04.4 and 19.10 64bit (instead of Raspbian 32-bit), for Mattermost Team server.
That’s the fix!! Don’t use Raspbian 32-bit. Use Ubuntu 18.04.4 64bit, or 19.10 64-bit.
There’s something about 32bit Raspbian which is slow, for Mattermost. But 64bit Ubuntu 18.04.4 (or 19.10) rips way, way faster on an image upload. The server is totally usable in a daily driver sort of way!!
How much faster?
Here’s a test I did. I ran Mattermost Team Server in a Debian 10.3 64-bit VM (ARM64). I gave the VM only one of my CPU cores, non-hyperthreaded (SMT is disabled). This is an i5 Intel Core, 8th gen. A laptop, Asus UX330U. Then I uploaded a 4.0 MB image. The thumbnail-processing with maxed-out CPU took just under 4 seconds (just timed on a wristwatch). Then I uploaded that same 4.0MB image into the RPi4. It also took just under 4 seconds (utilizing all 4 cores). Like it was identical performance! This is a night and day difference, going from 32-bit to 64-bit. This is going from minutes and minutes, down to under 4 seconds. That 4.0MB image would have crippled the RPi4 in 32bit Raspbian!
Problem solved! No code changes needed!
Hi @esbeeb
As mentioned I replicated your result with ubuntu server 18.04 and 64 bit mm code running on a Pi3 with very acceptable performance.
I then got to wondering. Raspbian now has a 64 bit kernel but apparently all userland code is stuck at 32 bit, however - plenty of time atm 
So - I have my pi3 running raspbian buster with a mm instance. Did a update/upgrade and then added “arm_64bit=1” to config.txt in /boot
After reboot it’s using the arm8 kernel.
Fire up existing 32bit mm instance and upload my test image (1.46mb) and that still takes well over 30 seconds to process.
I then simply copied the 64bit bin/mattermost executable from my ubuntu 18.04 instance onto the raspiban pi3 and re-ran the test - that image loaded in under 5 seconds which is more or less identical to ubuntu full blown 64 bit…
So - this is all fixed for stock Rasbian with one line of config and the 64 bit arm image…
Very happy bunny now
Sweet find. That opens the door to Raspbian… you just have to shoehorn 64-bitness in there.
Complications can arise, however, from such a strange combination of 64bit and 32bit. For example, I just tried what you suggested, and now installing/running Wireguard became a nightmare (as in, Wireguard is broken). I can’t even compile Wireguard from source, without a build-system battle on my hands.
At least with Ubuntu 64bit, Wireguard installed and ran like a breeze.
OK - as a rule I try to avoid compiling so didn’t see that. As noted previously there is also the issue with total plugin incompatibility as I haven’t been able to find ARM versions, so if any are required/desired it’s x64 only.
Given this and your comment on general “mixing” issues I’m going to abandon my plan to migrate my main Pi4 server to 64 bit at least until a proper 64 bit Raspian release for all supported packages.
A pain as I’ll have to run MM on either the Ubuntu Pi3 or leave it on the NUC if I want plugins. Until I found “/remind” there weren’t any I wanted/needed - now I want one…
I’ll call this a moral (pyrrhic?) victory for now