Upgraded and version number does not update

Troubleshooting
1

I was attempting to upgrade from 5.18.0 to 5.31.0. by following the steps in Redirect.

I first tried to follow the steps to go directly to 5.31.0 but the mattermost server would not start. Then I noticed the Note that said rolling upgrades can only be done by one minor version number at a time. So I tried 5.19.3.

The steps were as follows:

  1. cd /tmp

  2. wget https://releases.mattermost.com/5.19.3/mattermost-5.19.3-linux-amd64.tar.gz

  3. tar -xf mattermost*.gz --transform=‘s,[1]+,\0-upgrade,’

  4. systemctl stop mattermost

  5. cd /opt

  6. find mattermost/ mattermost/client/ -mindepth 1 -maxdepth 1 ! ( -type d ( -path mattermost/client -o -path mattermost/client/plugins -o -path mattermost/config -o -path mattermost/logs -o -path mattermost/plugins -o -path mattermost/data ) -prune ) | sort | sudo xargs echo rm -r

  7. chown -hR www-data:www-data /tmp/mattermost-upgrade/

  8. cp -an /tmp/mattermost-upgrade/. mattermost/

  9. rm -r /tmp/mattermost-upgrade/

  10. rm /tmp/mattermost*.gz

  11. systemctl start mattermost

  12. /opt/mattermost/bin/mattermost version

It seems to go smoothly but the results of 12. are:

Version: 5.18.0
Build Number: 5.18.0
Build Date: Tue Dec 17 05:01:10 UTC 2019
Build Hash: cd38d63bf448ae791f252c3704a705e94b26959f
Build Enterprise Ready: true
DB Version: 5.18.0

Any ideas on why the version number did not update to 5.19.3?

Can I upgrade directly to 5.31.0? If I must do it by each minor version I think I can write a bash script to do it.

Thanks,
Dennis

  1. ^/ ↩︎

2

Hello, @dworthem

I have this cheat sheet on my end that I use for upgrades in general. Can you please give it a try?

wget https://releases.mattermost.com/5.23.0/mattermost-5.31.0-linux-amd64.tar.gz
tar -xzf mattermost*.gz
cp -R /opt/mattermost/config/* /tmp/mattermost/config/
sudo su
mv /opt/mattermost/data /tmp/mattermost/data
rsync -au /opt/mattermost/plugins/ /tmp/mattermost/plugins
rsync -au /opt/mattermost/client/plugins/ /tmp/mattermost/client/plugins
systemctl stop mattermost
mv /opt/mattermost /opt/old.mattermost
mv /tmp/mattermost /opt/mattermost
chown -R mattermost:mattermost /opt/mattermost
systemctl start mattermost
**Confirm Mattermost is working**
rm /tmp/mattermost*.gz
rm -rf /opt/old.mattermost/

Please make sure that you generate a backup of the instance before you proceed with the upgrade. Let me know how it goes on your end.

3

Excellent it worked fine. I went from 5.18.0 to 5.19.3 using wget https://releases.mattermost.com/5.19.3/mattermost-5.19.3-linux-amd64.tar.gz

Is there a way using these steps to go directly from 5.19.3 to 5.31.0 or do I have to go one minor version at a time?

Mattermost should put your instructions in place of those in https://docs.mattermost.com/administration/upgrade.html#upgrading-to-the-latest-version.

Thanks much,
Dennis

1 Like
4

Hello, @dworthem

You are most welcome. Glad to know that it works.

There should not be an issue with upgrading to 5.19.3 to the latest one using the same set of steps. Just be aware of the upgrade notes that I would recommend you to check on besides the Mattermost changelog to understand the new features, bug fixes, and known issues on any particular server version.

5

I made a batch file and I executed it one version at a time and it worked through 5.29.2. After I applied 5.30.2, mattermost would not start and after applying 5.31.0 it still would not start. I restored my 5.18.0 directories and tried applying 5.31.0 directly and it still would not start. Did something big change after 5.29.2?

6

Appears to be a problem with plugins:
{“level”:“info”,“ts”:1611547889.5250523,“caller”:“app/server.go:225”,“msg”:“Server is initializing…”}
{“level”:“info”,“ts”:1611547889.525193,“caller”:“app/web_hub.go:83”,“msg”:“Starting websocket hubs”,“number_of_hubs”:8}
{“level”:“info”,“ts”:1611547889.5264525,“caller”:“utils/i18n.go:90”,“msg”:“Loaded system translations”,“for locale”:“en”,“from locale”:"/opt/mattermost/i18n/en.json"}
{“level”:“info”,“ts”:1611547889.5308564,“caller”:“sqlstore/supplier.go:246”,“msg”:“Pinging SQL”,“database”:“master”}
{“level”:“warn”,“ts”:1611547889.5900264,“caller”:“sqlstore/upgrade.go:137”,“msg”:“The database schema version and model versions do not match”,“schema_version”:“5.31.0”,“model_version”:“5.30.0”}
{“level”:“info”,“ts”:1611547889.7589,“caller”:“app/license.go:54”,“msg”:“License key from https://mattermost.com required to unlock enterprise features.”}
{“level”:“info”,“ts”:1611547889.7644029,“caller”:“sqlstore/post_store.go:1687”,“msg”:“Post.Message has size restrictions”,“max_characters”:16383,“max_bytes”:65535}
{“level”:“info”,“ts”:1611547891.6022644,“caller”:“app/server.go:474”,“msg”:“Current version is 5.30.0 (5.31.0/Fri Jan 15 02:36:16 UTC 2021/5c6f37ee6db78c8440145a27320baebdcc6e8311/c1b777c8d71a962500f64ddf8be9dd4df9b137f4)”,“current_version”:“5.30.0”,“build_number”:“5.31.0”,“build_date”:“Fri Jan 15 02:36:16 UTC 2021”,“build_hash”:“5c6f37ee6db78c8440145a27320baebdcc6e8311”,“build_hash_enterprise”:“c1b777c8d71a962500f64ddf8be9dd4df9b137f4”}
{“level”:“info”,“ts”:1611547891.602362,“caller”:“app/server.go:483”,“msg”:“Enterprise Build”,“enterprise_build”:true}
{“level”:“info”,“ts”:1611547891.6024017,“caller”:“app/server.go:489”,“msg”:“Printing current working”,“directory”:"/tmp"}
{“level”:“info”,“ts”:1611547891.6024282,“caller”:“app/server.go:490”,“msg”:“Loaded config”,“source”:“file:///opt/mattermost/config/config.json”}
{“level”:“info”,“ts”:1611547891.6181386,“caller”:“bleveengine/bleve.go:267”,“msg”:“UpdateConf Bleve”}
{“level”:“info”,“ts”:1611547891.6218698,“caller”:“app/license.go:54”,“msg”:“License key from https://mattermost.com required to unlock enterprise features.”}
{“level”:“info”,“ts”:1611547891.632153,“caller”:“mlog/log.go:218”,“msg”:“Starting up plugins”}
{“level”:“error”,“ts”:1611547891.6322424,“caller”:“app/plugin.go:178”,“msg”:“Failed to start up plugins”,“error”:“mkdir ./client/plugins: no such file or directory”}
{“level”:“info”,“ts”:1611547891.632303,“caller”:“mlog/log.go:218”,“msg”:“Starting up plugins”}
{“level”:“error”,“ts”:1611547891.6323223,“caller”:“app/plugin.go:178”,“msg”:“Failed to start up plugins”,“error”:“mkdir ./client/plugins: no such file or directory”}

7

I fixed this by updating /etc/systemd/system/multi-user.target.wants/mattermost.service

8

Hi, @dworthem

Happy to know that you got it fixed on your end. Can you please confirm the specific update you performed on that file?

Is it related to the following for dedicated database server or something else?

  • replace the WantedBy=postgresql.service or WantedBy=mysql.service line in the [Install] section with WantedBy=multi-user.target

I just want to make sense of the solution you implemented on your end.

9

It was:
[Unit]
Description=Mattermost
After=network.target
After=mysql.service
Requires=mysql.service

[Service]
Type=notify
ExecStart=/opt/mattermost/bin/mattermost
TimeoutStartSec=3600
Restart=always
RestartSec=10
WorkingDirectory=/opt/mattermost
User=mattermost
Group=mattermost
LimitNOFILE=49152

[Install]
WantedBy=multi-user.target

I changed Requires=mysql.service to BindsTo=mysql.service and
User and Group both to www-data

It all works fine except when I execute /opt/mattermost/bin/mattermost version I see:

{“level”:“info”,“ts”:1611713738.405366,“caller”:“mlog/log.go:218”,“msg”:“Starting up plugins”}
{“level”:“error”,“ts”:1611713738.4054487,“caller”:“app/plugin.go:178”,“msg”:“Failed to start up plugins”,“error”:“mkdir ./client/plugins: no such file or directory”}
{“level”:“info”,“ts”:1611713738.4054956,“caller”:“mlog/log.go:218”,“msg”:“Starting up plugins”}
{“level”:“error”,“ts”:1611713738.4055123,“caller”:“app/plugin.go:178”,“msg”:“Failed to start up plugins”,“error”:“mkdir ./client/plugins: no such file or directory”}

Do you have an idea why am getting those error, ./client/plugins does exist?

10

Hi, @dworthem

Awesome. The error that you are getting is potentially due to the the server needing the write access to /opt/mattermost/client/plugins to unpack any webapp-related resources. Can I please check if your current configuration for the plugins client directory is set to the following?

ahmaddanial@mattermost:~$ cat /opt/mattermost/config/config.json | grep "ClientDirectory"
        "ClientDirectory": "./client/plugins",

If yes, please change the ClientDirectory in config.json to the following and observe if it helps to solve the problem:

/opt/mattermost/client/plugins
11

I did so and I executed /opt/mattermost/bin/mattermost version again and now I see( I cut some of it out to avoid links:

{“level”:“info”,“ts”:1611716103.277966,“caller”:“mlog/log.go:218”,“msg”:“Starting up plugins”}
{“level”:“info”,“ts”:1611716103.2781806,“caller”:“app/plugin.go:223”,“msg”:“Syncing plugins from the file store”}
{“level”:“info”,“ts”:1611716104.7282743,“caller”:“mlog/sugar.go:19”,“msg”:“Ensuring Surveybot exists”,“plugin_id”:“com.mattermost.nps”}
{“level”:“error”,“ts”:1611716104.9196103,“caller”:“app/plugin.go:831”,“msg”:“Failed to install prepackaged plugin”,“path”:"/opt/mattermost/prepackaged_plugins/mattermost-plugin-incident-management-v1.2.0-linux-amd64.tar.gz",“error”:“Failed to install extracted prepackaged plugin /opt/mattermost/prepackaged_plugins/mattermost-plugin-incident-management-v1.2.0-linux-amd64.tar.gz: installExtractedPlugin: Unable to restart plugin on upgrade., a valid Mattermost Enterprise E20 license is required to use this plugin”,“errorVerbose”:"installExtractedPlugin: Unable to restart plugin on upgrade., a valid Mattermost Enterprise E20 license is required to use this plugin
{“level”:“error”,“ts”:1611716104.956229,“caller”:“mlog/log.go:232”,“msg”:“Unable to activate plugin”,“plugin_id”:“com.mattermost.plugin-incident-management”,“error”:“a valid Mattermost Enterprise E20 license is required to use this plugin”}
{“level”:“error”,“ts”:1611716104.990003,“caller”:“mlog/log.go:232”,“msg”:“Unable to activate plugin”,“plugin_id”:“com.mattermost.plugin-incident-management”,“error”:“a valid Mattermost Enterprise E20 license is required to use this plugin”}

I guess these are now all valid errors as I do not have an E20 license?

It seems these issues should be on the Mattermost documentation pages for upgrade.

12

Hi, @dworthem

I can see that the error is pointing to the Incident Management plugin - incident-management. With that being the case, can you please have it disabled and kick back the service again?

        "PluginStates": {
            "com.mattermost.nps": {
                "Enable": true
            },
            "com.mattermost.plugin-channel-export": {
                "Enable": true
            },
            "com.mattermost.plugin-incident-management": {
                "Enable": false
            }
        },

I will discuss this internally with the team if the ERROR should not appear in the first place or change the level for non-E20 users but for now, please confirm if disabling the plugin works.

13

I disabled the plugin and the error is gone. Did plugins start with version 5.30.2?

14

It looks like plugins functionality has been in place for a long time. I still don’t understand why the upgrade process was not a problem until 5.30.2

15

Hi, @dworthem

We have a bug report created for this - [MM-32387] Server app is trying to install E20 required plugins (e.g. incident management) on non-E20 installations - Mattermost

Feel free to join the discussion there so we primarily have this thread focusing on the issue with the inconsistent upgrade number. Thanks.

16

also I am curious why the version is 5.30.0 but the Build Number and DB Number are 5.31.0?

Version: 5.30.0
Build Number: 5.31.0
Build Date: Fri Jan 15 02:36:16 UTC 2021
Build Hash: 5c6f37ee6db78c8440145a27320baebdcc6e8311
Build Enterprise Ready: true
DB Version: 5.31.0

17

Not sure what the -h is in step 7. Below is a bash script that I use that works every time.

#! /bin/bash
# WARNING this script should not be ran unless a snapshot is taken of the server and database is backed up
# usage sudo ./upgrade_mattermost x.x.x
# x.x.x indicating the version number
set -e
set -x

cd /tmp
#check to ensure user entered version argument
if [ "$#" == "0" ]
then
    echo "Missing version number"
    exit 1
fi
upgrade_url="https://releases.mattermost.com/$1/mattermost-$1-linux-amd64.tar.gz"
response=$(wget -q "$upgrade_url")
if [ $? -ne 0 ]; then
    echo "failure to download upgrade files, check version number and format is [x.x.x]"
    exit 1
fi
wget $upgrade_url

#unzip and rename files so names do not conflict with current installation files
tar -xf mattermost*.gz --transform='s,^[^/]\+,\0-upgrade,'
echo STOPPING MATTERMOST SERVICE
systemctl stop mattermost

#first moves into mattermost installation dir then removes all files
cd /opt
find mattermost/ mattermost/client/ -mindepth 1 -maxdepth 1 \! \( -type d \( -path mattermost/client -o -path mattermost/client/plugins -o -path mattermost/config -o -path mattermost/logs -o -path mattermost/plugins -o -path mattermost/data \) -prune \) | sort | xargs rm -r

# COPYING UPGRADE FILES AND CONFIGURING GROUP OWNERSHIP
cp -an /tmp/mattermost-upgrade/. mattermost/
chown -R mattermost:mattermost mattermost
setcap cap_net_bind_service=+ep ./mattermost/bin/mattermost

echo STARTING MATTERMOST SERVICE - this may take a while
sleep 2 # for some reason the service will not start without a delay
systemctl start mattermost

# removes old files from the upgrade download
rm -rf /tmp/mattermost-upgrade/
rm -if /tmp/mattermost*.gz
echo SUCCESS ---- UPGRADE COMPLETE
1 Like
18

Hi @lramsey and weldcome to the Mattermost forums!

Thanks for providing this script!
The -h parameter to chown is used when you want to change the owner of symlinks (which is not possible on all file- and operating systems):

# chown --help
[...]
  -h, --no-dereference   affect symbolic links instead of any referenced file
                         (useful only on systems that can change the
                         ownership of a symlink)