The main mattermost config file gets reset to default values after upgrade to 5.31
Steps to reproduce
Stop mattermost server
Upgrade mattermost 5.29.1 to 5.31
Copy good config to ./config/config.json
Start mattermost server
Check config file: all settings have reverted to default (including DB, site URL, data storage location, etc)
Expected behavior
Mattermost 5.31 should just start and update the config.json with any new settings it expects to find in there.
Observed behavior
(unrelated to the issue but added for completeness: Our mattermost server crashed / stopped responding over the weekend. After a reboot of the VPS the MM server did not start up. I decided to update to run the planned update (5.29.1 to 5.31.0) as an attempt to fix the issue. Config.json was still good before the upgrade)
When mattermost 5.31 starts up the config file gets wiped. It obviously works to some degree (I can log in - it wouldn’t know my user if it didn’t have access to the database) but then the config file gets wiped. I have tried merging our live config (from 5.29.1) with the blank config (created by 5.31) to make sure we have valid json.
The config obviously works for startup (login works).
I thought I got it to work before but then after a while the config reset again (“Please configure your site URL on the System Console” popped up, files / embedded images in posts stopped working).
Any help appreciated.
I can’t attach a file here but will add another post with both config versions if it would be helpful.
Can you please confirm that the once the upgraded 5.31 server starts up, your config.json gets reverted to the default values? Eg. DataSource and LdapSettings?
I attempted to reproduce it on my end on a vanilla 5.29.0 upgraded to 5.31 and it was not reproducible.
May I know how did you upgrade your instance? Any specific documentation you referred to?
Hi, thanks for your response. Yes what you are describing matches my experience from earlier today. Though it did not always reset the config immediately upon start. One time it took a while (20+ minutes).
My initial reaction was “the script must have failed copying the config file back” (which makes no sense as my login still worked, without the correct config file there would not have been database access for logging in users)
#!/bin/bash
#steps taken from
# https://docs.mattermost.com/administration/upgrade.html
# last checked 2020-01-20
###
# You should gather the following information before starting the upgrade:
#
# Existing install directory - {install-path}
# If you don’t know where Mattermost Server is installed, use the whereis mattermost command. The output should be similar to /opt/mattermost/bin/mattermost. The install directory is everything before the first occurrence of the string /mattermost. In this example, the {install-path} is /opt. If that command does not produce any results because your version is older, try whereis platform instead.
#
# Location of your local storage directory
# The local storage directory contains all the files that users have attached to their messages. If you don’t know its location, open the System Console and go to Files > Storage in prior versions or Environment> File Storage in versions after 5.12 and read the value in Local Storage Directory. Relative paths are relative to the mattermost directory. For example, if the local storage directory is ./data/ then the absolute path is {install-path}/mattermost/data.
###
# here: $install_path and $data_path
#
version=$1
ts=$(date +'%F-%H-%M')
if test "$1" == ""; then
echo 'no version'
exit 1
else
test "$version" == "latest" && url=$(wget -qO- https://mattermost.com/download/ | grep 'https://[^&]*releases[^&]*mattermost[^&]*\.tar\.gz' -o | grep -v 'desktop' | uniq) || url="https://releases.mattermost.com/${version}/mattermost-${version}-linux-amd64.tar.gz"
tmp=${url#*/*/*/}
v=${tmp%%/*}
while true; do
read -p "Current version is [ $( grep -o 'Current version is [^[:space:]]*' /var/log/mattermost/mattermost.log | tail -n1 | cut -d' ' -f4 ) ]. Updating to [ $v ]. Do you wish to continue? " yn
case $yn in
[Yy]* ) break;;
[Nn]* ) exit;;
* ) echo "Please answer yes or no.";;
esac
done
install_path=/opt
mm_path=${install_path}/mattermost
data_path=${mm_path}/data/
###
# modified from MM instructions: extract to install_path
#cd /tmp/
wget "$url"
version=$(echo "$url" | cut -d/ -f4)
backupvers=mm-backup-pre-${version//./-}
here=$(pwd)
file=$(basename "$url")
mkdir "./$version"
#cd "./$version"
#tar -xf mattermost*.gz --transform='s,^[^/]\+,\0-upgrade,'
tar -C ./$version -xf $file
cd ${install_path}
read -p "READY TO STOP MM?"
# already in install_path (changed from MM instructions)
#cd ${install_path}
sudo systemctl stop mattermost || echo "Unable to stop MM"
echo "$(date) start DB SNAPSHOT now:"
aws rds create-db-snapshot --db-snapshot-identifier "${backupvers}" --db-instance-identifier "mattermost" --region "ap-southeast-2"
# MM Upgrade page suggestion:
#sudo cp -ra mattermost/ mattermost-$backupvers-$(date +'%F-%H-%M')/
#sudo find mattermost/ mattermost/client/ -mindepth 1 -maxdepth 1 \! \( -type d \( -path mattermost/client -o -path mattermost/client/plugins -o -path mattermost/config -o -path mattermost/logs -o -path mattermost/plugins -o -path mattermost/data \) -prune \) | sort | sudo xargs echo rm -r
# Quicker implementation. Less risk of losing data.
sudo mv -iv mattermost mattermost-$backupvers-$ts
sudo mv -iv $here/$version/mattermost ./mattermost
sudo chown -hR mattermost:mattermost ./mattermost
sudo cp -ra ./mattermost-$backupvers-$ts/logs ./mattermost/
sudo cp -ra ./mattermost-$backupvers-$ts/config ./mattermost/
sudo setcap cap_net_bind_service=+ep $mm_path/bin/mattermost
echo "$( date ) waiting for snapshot to complete"
aws rds wait db-snapshot-completed --db-instance-identifier "mattermost" --region "ap-southeast-2"
echo "$( date ) snapshot completed. Starting Mattermost"
#read -p "READY TO START MM AGAIN? Snapshot complete etc?"
sudo systemctl start mattermost
RES=$?
if test "$RES" -eq "0"; then
echo "$( date ) Mattermost started [ $RES ]"
else
echo ' **************************************************************************** '
echo ' ** HELP ****** HELP ****** HELP ****** HELP ****** HELP ****** HELP ******** '
echo ' ******** HELP ****** HELP ****** HELP ****** HELP ****** HELP ****** HELP ** '
echo ' **************************************************************************** '
read -p "$(date) UNABLE TO START MM. PLEASE CHECK. [ $RES ]"
fi
echo ' **************************************************************************** '
echo ' **************************************************************************** '
echo "$( date ) Open the System Console and change a setting, then revert it. This should enable the Save button for that page. SAVE, then reload that page"
echo "https://mm.snapit.group:40404/admin_console or https://mm.snapit.group:40404/admin_console/site_config/localization"
while true; do
read -p "Have you updated MM settings and wish to continue? " yn
case $yn in
[Yy]* ) break;;
[Nn]* ) exit;;
* ) echo "Please answer yes or no.";;
esac
done
echo "$( date ) config updated. Stopping Mattermost"
sudo systemctl stop mattermost
RES=$?
if test "$RES" -eq "0"; then
echo "$( date ) Mattermost stopped again [ $RES ]"
else
echo ' **************************************************************************** '
echo ' ** HELP ****** HELP ****** HELP ****** HELP ****** HELP ****** HELP ******** '
echo ' ******** HELP ****** HELP ****** HELP ****** HELP ****** HELP ****** HELP ** '
echo ' **************************************************************************** '
read -p "$(date) UNABLE TO STOP MM. PLEASE CHECK. [ $RES ]"
fi
echo "$( date ) MM Stopped. Copy plugins back"
sudo cp -ra ./mattermost-$backupvers-$ts/plugins ./mattermost/
echo "$( date ) Copy client plugins back"
sudo cp -ra ./mattermost-$backupvers-$ts/client/plugins ./mattermost/client/
echo "$( date ) All plugins copied. Starting Mattermost"
sudo systemctl start mattermost
RES=$?
if test "$RES" -eq "0"; then
echo "$( date ) Mattermost started again - all set and upgraded [ $RES ]"
else
echo ' **************************************************************************** '
echo ' ** HELP ****** HELP ****** HELP ****** HELP ****** HELP ****** HELP ******** '
echo ' ******** HELP ****** HELP ****** HELP ****** HELP ****** HELP ****** HELP ** '
echo ' **************************************************************************** '
read -p "$(date) UNABLE TO START MM. PLEASE CHECK. [ $RES ]"
fi
cd $here
while true; do
read -p "Do you want to proceed with a system update? " yn
case $yn in
[Yy]* ) break;;
[Nn]* ) exit;;
* ) echo "Please answer yes or no.";;
esac
done
sudo apt update
sudo apt upgrade
sudo apt autoremove
sudo apt autoclean
sudo apt clean
while true; do
read -p "Do you want to proceed with a system reboot? " yn
case $yn in
[Yy]* ) break;;
[Nn]* ) exit;;
* ) echo "Please answer yes or no.";;
esac
done
sudo reboot now
fi
thanks all, yes that github issue looks like the exact issue that has been happening here.
I am pretty confident that disabling the Enterprise-only plugins fixed it for me. (At the same time as disabling the plugins I also disabled a pretty crude version check which may have contributed to the rollback but I am not too keen on verifying this on my live system - sudo /opt/mattermost/bin/mattermost version 2>/dev/null | grep "Build Number:" | cut -d: -f2' )
Either way, rolling the config file back to defaults is quite a significant issue, would be good to see a note on this under “Known Issues” on the update page.
It is this version check which is causing the rollback. You are running the binary as root, which has a different home directory. Therefore, it will take the default config.json and reset that.
The fix should be to run the version check as the same user which is running the mattermost service.
This has not been an issue in previous versions of mattermost, so I am a bit surprised this has become a problem now as the root user in my case does have read access to the config file and the rolled back version is owned by mattermost:mattermost just like the original version was (with identical permissions). Again, a note that the config file may roll back “randomly” on some systems would be great in the “important release notes” or “known issues” documents.
Either way, I saw a note that there is a new way to get the mattermost version and I will try that at some point for my scripts.
The CLI commands using the binary use an old and brittle way to get information. That’s why there are lot of issues in general. It is suggested to use mmctl (which is the new way you mentioned) for any CLI purposes to interact with Mattermost.
It looks like I was a bit hasty in my analysis earlier. My apologies for that. I tried to reproduce this by running the binary as root, but I don’t see any config reset behavior.
The reason I said that would be the issue was because we had another customer report the exact same thing that running the “mattermost version” CLI as a different user was what reverted the config for them.
Thanks for looking into it again @agnivade - I can confirm the version check (as root) had been running successfully several times per day for about 6 weeks on version 5.29.
The config reset issue only began after upgrading from 5.29 to 5.31 (we skipped 5.30).
For what it’s worth, mattermost reset the config file back to defaults again after updating to 5.32.1 yesterday.
After the update I start sudo systemctl start mattermost without plugins so I can change/revert/save settings in the system console (as per upgrade instructions).
Before touching anything, I notice the banner “please set site name” (or something along those lines) at the top, indicating that the config file has been wiped. On inspection in the file system, I can confirm config.json has been reset to defaults.
I have merged the blank config created by mattermost with my previous file and it has been working since then.