Upgrade MM 5.3.1 to latest version fails

SuziQ · February 5, 2021, 9:47am

Hello,

alright, the files can all be opened again. I had moved the data directory to /opt/mattermost.data, but in the backup of the config.json file it still said ./data/. That was my mistake.

I now have a correct MM 5.3.1 installation again and will soon try to update to version 5.9.8 ESR as suggested by @ahmaddanial.
→ https://releases.mattermost.com/5.9.8/mattermost-5.9.8-linux-amd64.tar.gz?src=arc

SuziQ · February 5, 2021, 12:23pm

Well, I’ve created mattermost.service again from
→ <no title> — Mattermost documentation

My /lib/systemd/system/mattermost.service now (values from above):

[Unit]
Description=Mattermost
After=network.target
After=mysql.service
BindsTo=mysql.service

[Service]
Type=notify
ExecStart=/opt/mattermost/bin/mattermost
TimeoutStartSec=3600
Restart=always
RestartSec=10
WorkingDirectory=/opt/mattermost
User=mattermost
Group=mattermost
LimitNOFILE=49152

[Install]
WantedBy=mysql.service

My MM version 5.3.1 is already running

SuziQ · February 5, 2021, 12:29pm

Then I’ve upgraded to version 5.9.8 ESR with these steps:

get MM version 5.9.8

cd /opt/tmp
wget https://releases.mattermost.com/5.9.8/mattermost-5.9.8-linux-amd64.tar.gz
tar -xzf mattermost-5.9.8-linux-amd64.tar.gz

cd /opt

backup current state

systemctl stop mattermost
mysqldump -u root -p mattermost > mm.sql
rsync -a mattermost mm.bak
rsync -a mattermost.data mm.data.bak

prepare new MM version

rsync -a mattermost/config/ tmp/mattermost/config/
rsync -au mattermost/plugins/ tmp/mattermost/plugins
rsync -au mattermost/client/plugins/ tmp/mattermost/client/plugins

mv mattermost old.mattermost
mv tmp/mattermost mattermost
chown -R mattermost:mattermost mattermost

SuziQ · February 5, 2021, 12:37pm

After starting mattermost.service there’s shown an error, but the service ist already running for some seconds as you can see at first ps -ef command.

systemctl start mattermost

Job for mattermost.service failed because the control process exited with error code. See “systemctl status mattermost.service” and “journalctl -xe” for details.

systemctl status mattermost

● mattermost.service - Mattermost
Loaded: loaded (/lib/systemd/system/mattermost.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Fre 2021-02-05 13:10:24 CET; 2s ago
Process: 27369 ExecStart=/opt/mattermost/bin/mattermost (code=exited, status=119)
Main PID: 27369 (code=exited, status=119)

Feb 05 13:10:24 SERVER systemd[1]: Failed to start Mattermost.
Feb 05 13:10:24 SERVER systemd[1]: mattermost.service: Unit entered failed state.
Feb 05 13:10:24 SERVER systemd[1]: mattermost.service: Failed with result ‘exit-code’.
root@SERVER:/opt#
root@SERVER:/opt#
root@SERVER:/opt#
root@SERVER:/opt# systemctl status mattermost
● mattermost.service - Mattermost
Loaded: loaded (/lib/systemd/system/mattermost.service; enabled; vendor preset: enabled)
Active: activating (start) since Fre 2021-02-05 13:10:45 CET; 937ms ago
Main PID: 27407 (mattermost)
Tasks: 11 (limit: 65000)
Memory: 27.7M
CPU: 353ms
CGroup: /system.slice/mattermost.service
└─27407 /opt/mattermost/bin/mattermost

Feb 05 13:10:45 SERVER systemd[1]: mattermost.service: Service hold-off time over, scheduling restart.
Feb 05 13:10:45 SERVER systemd[1]: Stopped Mattermost.
Feb 05 13:10:45 SERVER systemd[1]: Starting Mattermost…
Feb 05 13:10:45 SERVER mattermost[27407]: {“level”:“info”,“ts”:1612527045.976185,“caller”:“utils/i18n.go:83”,“msg”:"Loaded system
Feb 05 13:10:45 SERVER mattermost[27407]: {“level”:“info”,“ts”:1612527045.9765236,“caller”:“app/server_app_adapters.go:58”,“msg”:
Feb 05 13:10:45 SERVER mattermost[27407]: {“level”:“info”,“ts”:1612527045.9883635,“caller”:“sqlstore/supplier.go:215”,“msg”:"Ping
Feb 05 13:10:46 SERVER mattermost[27407]: {“level”:“error”,“ts”:1612527046.0502627,“caller”:“sqlstore/supplier.go:807”,“msg”:"Fai
root@SERVER:/opt#
root@SERVER:/opt#
root@SERVER:/opt#
root@SERVER:/opt# ps -ef | grep -i mattermost
matterm+ 27425 1 31 13:10 ? 00:00:00 /opt/mattermost/bin/mattermost
root 27439 21018 0 13:10 pts/0 00:00:00 grep --color=auto -i mattermost
root@SERVER:/opt#
root@SERVER:/opt#
root@SERVER:/opt# ps -ef | grep -i mattermost
root 27459 21018 0 13:11 pts/0 00:00:00 grep --color=auto -i mattermost

Ok, return to v5.3.1 …

rm -r mattermost
mv old.mattermost mattermost
mysql> drop database mattermost;
mysql> create database mattermost;
mysql -u root -p mattermost < mm.sql
systemctl start mattermost
systemctl status mattermost
● mattermost.service - Mattermost
Loaded: loaded (/lib/systemd/system/mattermost.service; enabled; vendor preset: enabled)
Active: active (running) since Fre 2021-02-05 13:39:48 CET; 17min ago
Main PID: 30413 (mattermost)
Tasks: 23 (limit: 65000)
Memory: 78.7M
CPU: 2.676s
CGroup: /system.slice/mattermost.service
├─30413 /opt/mattermost/bin/mattermost
└─30481 plugins/jira/plugin.exe

Voila! 5.3.1 is up and running …

XxLilBoPeepsxX · February 7, 2021, 6:43pm

I am so sorry for the delay in response, work has been hectic for me lately, to say the least, and yes we definitely can work to restore the problem of the unopenable files first, since that pretty much deadlocks us from even attempting to fix the database schema issues in itself.

XxLilBoPeepsxX · February 7, 2021, 7:00pm

I copied and pasted your command sequences you provided into a code formatting below, just because it is much easier to read and process for me, and as such probably is for others too.

#systemctl start mattermost

Job for mattermost.service failed because the control process exited with error code. See "systemctl status mattermost.service" and "journalctl -xe" for details.

#systemctl status mattermost

● mattermost.service - Mattermost
   Loaded: loaded (/lib/systemd/system/mattermost.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Fre 2021-02-05 13:10:24 CET; 2s ago
  Process: 27369 ExecStart=/opt/mattermost/bin/mattermost (code=exited, status=119)
 Main PID: 27369 (code=exited, status=119)

Feb 05 13:10:24 SERVER systemd[1]: Failed to start Mattermost.
Feb 05 13:10:24 SERVER systemd[1]: mattermost.service: Unit entered failed state.
Feb 05 13:10:24 SERVER systemd[1]: mattermost.service: Failed with result 'exit-code'.

root@SERVER:/opt# systemctl status mattermost
● mattermost.service - Mattermost
   Loaded: loaded (/lib/systemd/system/mattermost.service; enabled; vendor preset: enabled)
   Active: activating (start) since Fre 2021-02-05 13:10:45 CET; 937ms ago
 Main PID: 27407 (mattermost)
    Tasks: 11 (limit: 65000)
   Memory: 27.7M
      CPU: 353ms
   CGroup: /system.slice/mattermost.service
           └─27407 /opt/mattermost/bin/mattermost

Feb 05 13:10:45 SERVER systemd[1]: mattermost.service: Service hold-off time over, scheduling restart.
Feb 05 13:10:45 SERVER systemd[1]: Stopped Mattermost.
Feb 05 13:10:45 SERVER systemd[1]: Starting Mattermost...
Feb 05 13:10:45 SERVER mattermost[27407]: {"level":"info","ts":1612527045.976185,"caller":"utils/i18n.go:83","msg":"Loaded system
Feb 05 13:10:45 SERVER mattermost[27407]: {"level":"info","ts":1612527045.9765236,"caller":"app/server_app_adapters.go:58","msg":
Feb 05 13:10:45 SERVER mattermost[27407]: {"level":"info","ts":1612527045.9883635,"caller":"sqlstore/supplier.go:215","msg":"Ping
Feb 05 13:10:46 SERVER mattermost[27407]: {"level":"error","ts":1612527046.0502627,"caller":"sqlstore/supplier.go:807","msg":"Fai

root@SERVER:/opt# ps -ef | grep -i mattermost
matterm+ 27425     1 31 13:10 ?        00:00:00 /opt/mattermost/bin/mattermost
root     27439 21018  0 13:10 pts/0    00:00:00 grep --color=auto -i mattermost

root@SERVER:/opt# ps -ef | grep -i mattermost
root     27459 21018  0 13:11 pts/0    00:00:00 grep --color=auto -i mattermost

Ok, return to v5.3.1 ... :frowning:

> rm -r mattermost
> mv old.mattermost mattermost
> mysql> drop database mattermost;
> mysql> create database mattermost;
> mysql -u root -p mattermost < mm.sql
> systemctl start mattermost
> systemctl status mattermost
    ● mattermost.service - Mattermost
       Loaded: loaded (/lib/systemd/system/mattermost.service; enabled; vendor preset: enabled)
       Active: active (running) since Fre 2021-02-05 13:39:48 CET; 17min ago
     Main PID: 30413 (mattermost)
        Tasks: 23 (limit: 65000)
       Memory: 78.7M
          CPU: 2.676s
       CGroup: /system.slice/mattermost.service
               ├─30413 /opt/mattermost/bin/mattermost

I noticed a few things based on the commands you were running here that could be causing potential conflicts when you are attempting to upgrade, mainly being that (unless you haven’t included it in the commands here) you aren’t stopping the Mattermost program before making upgrades, which is something you must do, especially when upgrading the binary itself. To do this, simply run the command sudo systemctl stop mattermost && systemctl status mattermost and then you can verify that it is stopped with the resulting output (which came from one of my own mattermost dev instances that I just stopped and pulled the output from)

root@ubuntu:~# systemctl stop mattermost && systemctl status mattermost
● mattermost.service - Mattermost
     Loaded: loaded (/lib/systemd/system/mattermost.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Sun 2021-02-07 13:56:35 EST; 40ms ago
    Process: 3190695 ExecStart=/opt/mattermost/bin/mattermost (code=exited, status=0/SUCCESS)
   Main PID: 3190695 (code=exited, status=0/SUCCESS)

Feb 07 07:08:07 ubuntu mattermost[3190695]: {"level":"info","ts":1612699686.6868248,"caller":"app/server.go:817","msg":>Feb 07 07:08:07 ubuntu mattermost[3190695]: {"level":"info","ts":1612699687.291837,"caller":"app/server.go:860","msg":">Feb 07 07:08:07 ubuntu mattermost[3190695]: {"level":"info","ts":1612699687.521241,"caller":"commands/server.go:106","m>Feb 07 07:08:08 ubuntu systemd[1]: Started Mattermost.
Feb 07 13:56:35 ubuntu systemd[1]: Stopping Mattermost...
Feb 07 13:56:35 ubuntu mattermost[3190695]: {"level":"info","ts":1612724195.3124132,"caller":"app/server.go:643","msg":>Feb 07 13:56:35 ubuntu mattermost[3190695]: {"level":"info","ts":1612724195.4578426,"caller":"app/web_hub.go:115","msg">Feb 07 13:56:35 ubuntu mattermost[3190695]: {"level":"info","ts":1612724195.7608464,"caller":"app/server.go:721","msg":>Feb 07 13:56:35 ubuntu systemd[1]: mattermost.service: Succeeded.
Feb 07 13:56:35 ubuntu systemd[1]: Stopped Mattermost.

Then once upgrading your instance, you must start the Mattermost server again with sudo systemctl start mattermost and hopefully it will get a bit further.

As a side note, I also noticed that while you’re providing the output of systemctl status mattermost the output of journalctl -xe contains much more detailed and specific error logs and would be a great resource to check immediately after encountering an error on start, for example. (checking immediately after encountering an error ensures that the most recent error log messages are shown first, which will be the ones your looking for)

SuziQ · February 9, 2021, 7:06pm

Hello, as I described above, all files can be opened again. I had taken the data directory out of the Mattermost installation (/opt/mattermost.data) and the config file backup was still set to the old path. This was my mistake. I HAVE a correctly running v5.3.1 - like before

SuziQ · February 9, 2021, 7:20pm

How can I design text just like you?

SuziQ · February 9, 2021, 7:26pm

I have also described my update procedure step by step above. I now have more than a dozen upgrade attempts behind me and I always stop the Mattermost service BEFORE exporting the database and before backing up of the installation directory.

SuziQ · February 9, 2021, 7:31pm

I will do that, but not this evening …
Thank you so very much for your persistence

XxLilBoPeepsxX · February 9, 2021, 8:20pm

You can code format the text by putting three backticks ``` on a separate line before and after the text you want to format.

I’ll be watching to see what you discover, thanks for being willing to work with us!

SuziQ · February 10, 2021, 8:08pm

Hello, XxLilBoPeepsxX,
I have done it now again . First my upgrade procedure, than the result:

get MM version 5.9.8

mkdir /opt/tmp
cd /opt/tmp
wget https://releases.mattermost.com/5.9.8/mattermost-5.9.8-linux-amd64.tar.gz
tar -xzf mattermost-5.9.8-linux-amd64.tar.gz

cd /opt

backup current MM state

systemctl stop mattermost
mysqldump -u root -p mattermost > mm.sql
rsync -a mattermost mm.bak
rsync -a mattermost.data mm.data.bak

prepare new MM version

rsync -a mattermost/config/ tmp/mattermost/config/
rsync -au mattermost/plugins/ tmp/mattermost/plugins
rsync -au mattermost/client/plugins/ tmp/mattermost/client/plugins

change MM installations

mv mattermost old.mattermost
mv tmp/mattermost mattermost
chown -R mattermost:mattermost mattermost

SuziQ · February 10, 2021, 8:10pm

start and check “mattermost.service”

root@SERVER:/opt# systemctl start mattermost
Job for mattermost.service failed because the control process exited with error code. See "systemctl status mattermost.service" and "journalctl -xe" for details.
root@SERVER:/opt#
root@SERVER:/opt#
root@SERVER:/opt# journalctl -xe
Feb 10 21:00:51 SERVER audit[4518]: AVC apparmor="DENIED" operation="open" profile="snap.rocketchat-server.rocketchat-mongo" name
Feb 10 21:00:52 SERVER kernel: audit_printk_skb: 6 callbacks suppressed
Feb 10 21:00:52 SERVER kernel: audit: type=1400 audit(1612987251.997:2302115): apparmor="DENIED" operation="open" profile="snap.r
Feb 10 21:00:52 SERVER kernel: audit: type=1400 audit(1612987251.997:2302116): apparmor="DENIED" operation="open" profile="snap.r
Feb 10 21:00:52 SERVER systemd[1]: mattermost.service: Service hold-off time over, scheduling restart.
Feb 10 21:00:52 SERVER systemd[1]: Stopped Mattermost.
-- Subject: Unit mattermost.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mattermost.service has finished shutting down.
Feb 10 21:00:52 SERVER systemd[1]: Starting Mattermost...
-- Subject: Unit mattermost.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mattermost.service has begun starting up.
Feb 10 21:00:52 SERVER mattermost[23320]: {"level":"info","ts":1612987252.479473,"caller":"utils/i18n.go:83","msg":"Loaded system
Feb 10 21:00:52 SERVER mattermost[23320]: {"level":"info","ts":1612987252.4797087,"caller":"app/server_app_adapters.go:58","msg":
Feb 10 21:00:52 SERVER mattermost[23320]: {"level":"info","ts":1612987252.4869711,"caller":"sqlstore/supplier.go:215","msg":"Ping
Feb 10 21:00:52 SERVER mattermost[23320]: {"level":"error","ts":1612987252.5463796,"caller":"sqlstore/supplier.go:807","msg":"Fai
Feb 10 21:00:52 SERVER audit[4518]: AVC apparmor="DENIED" operation="open" profile="snap.rocketchat-server.rocketchat-mongo" name
Feb 10 21:00:52 SERVER audit[4518]: AVC apparmor="DENIED" operation="open" profile="snap.rocketchat-server.rocketchat-mongo" name
Feb 10 21:00:53 SERVER kernel: audit: type=1400 audit(1612987252.997:2302117): apparmor="DENIED" operation="open" profile="snap.r
Feb 10 21:00:53 SERVER kernel: audit: type=1400 audit(1612987252.997:2302118): apparmor="DENIED" operation="open" profile="snap.r
Feb 10 21:00:53 SERVER systemd[1]: mattermost.service: Main process exited, code=exited, status=119/n/a
Feb 10 21:00:53 SERVER systemd[1]: Failed to start Mattermost.
-- Subject: Unit mattermost.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mattermost.service has failed.
-- 
-- The result is failed.
Feb 10 21:00:53 SERVER systemd[1]: mattermost.service: Unit entered failed state.
Feb 10 21:00:53 SERVER systemd[1]: mattermost.service: Failed with result 'exit-code'.
Feb 10 21:00:53 SERVER audit[4518]: AVC apparmor="DENIED" operation="open" profile="snap.rocketchat-server.rocketchat-mongo" name
Feb 10 21:00:54 SERVER kernel: audit: type=1400 audit(1612987253.997:2302119): apparmor="DENIED" operation="open" profile="snap.r
Feb 10 21:00:54 SERVER kernel: audit: type=1400 audit(1612987254.005:2302120): apparmor="DENIED" operation="open" profile="snap.r
Feb 10 21:00:54 SERVER audit[4518]: AVC apparmor="DENIED" operation="open" profile="snap.rocketchat-server.rocketchat-mongo" name
Feb 10 21:00:54 SERVER audit[4518]: AVC apparmor="DENIED" operation="open" profile="snap.rocketchat-server.rocketchat-mongo" name
Feb 10 21:00:54 SERVER audit[4518]: AVC apparmor="DENIED" operation="open" profile="snap.rocketchat-server.rocketchat-mongo" name
Feb 10 21:00:55 SERVER kernel: audit: type=1400 audit(1612987254.997:2302121): apparmor="DENIED" operation="open" profile="snap.r
Feb 10 21:00:55 SERVER kernel: audit: type=1400 audit(1612987254.997:2302122): apparmor="DENIED" operation="open" profile="snap.r
lines 1021-1064/1064 (END)

I hope so there’s some output you can see my upgrade error(s) … Good luck!

SuziQ · February 10, 2021, 8:31pm

My fall back procedure:

remove new version, restore old one

rm -r mattermost
mv old.mattermost mattermost

remove all database tables in db “mattermost”

mysql -u root -p
MariaDB [(none)]> drop database mattermost;
MariaDB [(none)]> create database mattermost;
MariaDB [(none)]> quit

restore previous database

mysql -u root -p mattermost < mm.sql

start and check “mattermost.service”

systemctl start mattermost
systemctl status mattermost

● mattermost.service - Mattermost
   Loaded: loaded (/lib/systemd/system/mattermost.service; enabled; vendor preset: enabled)
   Active: active (running) since Mit 2021-02-10 21:26:11 CET; 1min 7s ago
 Main PID: 25895 (mattermost)
    Tasks: 22 (limit: 65000)
   Memory: 84.5M
      CPU: 1.283s
   CGroup: /system.slice/mattermost.service
           ├─25895 /opt/mattermost/bin/mattermost
           └─25967 plugins/jira/plugin.exe

Feb 10 21:26:10 SERVER mattermost[25895]: {"level":"info","ts":1612988770.6392508,"caller":"app/app.go:636","msg":"Migrating emoj
Feb 10 21:26:10 SERVER mattermost[25895]: {"level":"info","ts":1612988770.6521726,"caller":"app/plugin.go:100","msg":"Starting up
Feb 10 21:26:11 SERVER mattermost[25895]: {"level":"info","ts":1612988771.3832166,"caller":"app/server.go:88","msg":"Starting Ser
Feb 10 21:26:11 SERVER mattermost[25895]: {"level":"info","ts":1612988771.3836057,"caller":"app/server.go:148","msg":"Server is l
Feb 10 21:26:11 SERVER mattermost[25895]: {"level":"info","ts":1612988771.5963547,"caller":"app/web_hub.go:75","msg":"Starting 4 
Feb 10 21:26:11 SERVER mattermost[25895]: {"level":"info","ts":1612988771.638402,"caller":"jobs/workers.go:62","msg":"Starting wo
Feb 10 21:26:11 SERVER mattermost[25895]: {"level":"info","ts":1612988771.6385572,"caller":"commands/server.go:265","msg":"Sendin
Feb 10 21:26:11 SERVER systemd[1]: Started Mattermost.
Feb 10 21:26:11 SERVER mattermost[25895]: {"level":"info","ts":1612988771.6425447,"caller":"jobs/schedulers.go:68","msg":"Startin
Feb 10 21:26:53 SERVER systemd[1]: Started Mattermost.

remove old backups

rm -r mm.*

XxLilBoPeepsxX · February 11, 2021, 8:33pm

SuziQ:

Feb 10 21:00:52 SERVER mattermost[23320]: {"level":"info","ts":1612987252.4869711,"caller":"sqlstore/supplier.go:215","msg":"Ping
Feb 10 21:00:52 SERVER mattermost[23320]: {"level":"error","ts":1612987252.5463796,"caller":"sqlstore/supplier.go:807","msg":"Fai

By any chance could you perhaps pull the rest of the error messages here? when you run the journalctl -xe command you should be able to use the arrow keys to navigate up and down the list, and well as scrolling the text left to right within the command prompt.

It appears from the initial log information here

(which although cut off, I have seen before) is indicating there is an error establishing the connection to the database, but until we see the rest of the error log string, we can’t be sure that this is indeed what the log message says, and rule out other potential database issues.

SuziQ · February 12, 2021, 9:46am

Oh God, I see my mistake! grrrrr
I have unfortunately not paid attention to it
Okay, the whole thing again … … …

SuziQ · February 12, 2021, 11:07am

Ready!
Ok, the ouput of journalctl -xe is much to long for my monitor and there’s NO horizintal scroll bar. So I’ve redirected the output to file journalctl-xe.txt. Because of the file size I wanted to attach this TXT file, but only pictures are allowed here. Renaming to PNG also fails. 1.100 lines are too much, only 32 MB are allowed here.
So please download it here (7 days):
http://wikisend.com/download/334716/journalctl-xe.txt

XxLilBoPeepsxX · February 19, 2021, 1:28pm

Good thing I checked the forum today! Got it with just a day left!

Even better, I found our error!

@ahmaddanial would you be able to suggest a reason that this error would be being caused? I just pulled this from the log file that @SuziQ was able to provide, it is as follows:

Feb 12 11:06:33 SERVER mattermost[15778]: {"level":"info","ts":1613124393.5221512,"caller":"sqlstore/supplier.go:215","msg":"Pinging SQL master database"}
Feb 12 11:06:33 SERVER mattermost[15778]: {"level":"error","ts":1613124393.5900004,"caller":"sqlstore/supplier.go:807","msg":"Failed to create index Error 1709: Index column size too large. The maximum column size is 767 bytes."}

The full error sequence can be seen as follows, which I derived after removing the AppArmor and other logging entries from the Journal logs that were provided.

-- Unit mattermost.service has begun starting up.
Feb 12 11:07:08 SERVER mattermost[15829]: {"level":"info","ts":1613124428.0044687,"caller":"utils/i18n.go:83","msg":"Loaded system translations for 'en' from '/opt/mattermost/i18n/en.json'"}
Feb 12 11:07:08 SERVER mattermost[15829]: {"level":"info","ts":1613124428.0046842,"caller":"app/server_app_adapters.go:58","msg":"Server is initializing..."}
Feb 12 11:07:08 SERVER mattermost[15829]: {"level":"info","ts":1613124428.0156324,"caller":"sqlstore/supplier.go:215","msg":"Pinging SQL master database"}
Feb 12 11:07:08 SERVER mattermost[15829]: {"level":"error","ts":1613124428.0851138,"caller":"sqlstore/supplier.go:807","msg":"Failed to create index Error 1709: Index column size too large. The maximum column size is 767 bytes."}
Feb 12 11:07:09 SERVER systemd[1]: mattermost.service: Main process exited, code=exited, status=119/n/a
Feb 12 11:07:09 SERVER systemd[1]: Failed to start Mattermost.
-- Subject: Unit mattermost.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mattermost.service has failed.
-- 
-- The result is failed.
Feb 12 11:07:09 SERVER systemd[1]: mattermost.service: Unit entered failed state.
Feb 12 11:07:09 SERVER systemd[1]: mattermost.service: Failed with result 'exit-code'.

I did some additional research and saw some information on MySQL forums related to this issue possibly being due to a engine-wide character encoding variable set to UTF-8, which would restrict Mattermost from using multiple variants of UTF-8 across some of it’s columns, from what I understand.

ahmaddanial · February 21, 2021, 10:52pm

Hi @XxLilBoPeepsxX @SuziQ

The Error 1709: Index column size too large. The maximum column size is 767 bytes. that we are seeing here happens when the table ROW_FORMAT is COMPRESSED making the table’s row too large mentioned in this link:

Why do I see a MySQL error message saying “The maximum column size is 767 bytes”?

What we can do here is to change the ROW_FORMAT to DYNAMIC by following the steps provided the following link:

MySQL: changing ROW FORMAT to DYNAMIC or COMPRESSED

As an alternative, ensure the default format is DYNAMIC before creating or restoring your back up database.

SET GLOBAL innodb_default_row_format=DYNAMIC;`

Let me know how that goes.

SuziQ · February 22, 2021, 7:49pm

Good evening,
thanks for the links and instructions, but unfortunately still no success.

What I did:

mysql -u root -p

I changed all tables of database mattermost manually, because there are not really many tables.

MariaDB [(none)]> use mattermost;
MariaDB [(mattermost)]> ALTER TABLE <each table extra> ROW_FORMAT=DYNAMIC;

No error occurred and apparently only a few tables were changed.
Here is the recording:

MariaDB [mattermost]> ALTER TABLE Audits ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.50 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE ChannelMemberHistory ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.06 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE ChannelMembers ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.10 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE Channels ROW_FORMAT=DYNAMIC;
Query OK, 100 rows affected (0.28 sec)             
Records: 100  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE ClusterDiscovery ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.04 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE CommandWebhooks ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.02 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE Commands ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.04 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE Compliances ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.02 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE Emoji ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.05 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE FileInfo ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.15 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE IncomingWebhooks ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.07 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE Jobs ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.05 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE Licenses ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.09 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE OAuthAccessData ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.09 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE OAuthApps ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.07 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE OAuthAuthData ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.12 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE OutgoingWebhooks ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.06 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE PluginKeyValueStore ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.03 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE Posts ROW_FORMAT=DYNAMIC;
Query OK, 7059 rows affected (2.96 sec)                
Records: 7059  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE Preferences ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.17 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE Reactions ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.07 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE Roles ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.05 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE Schemes ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.05 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE Sessions ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.06 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE Status ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.04 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE Systems ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.07 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE TeamMembers ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.08 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE Teams ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.17 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE UserAccessTokens ROW_FORMAT=DYNAMIC;
Query OK, 0 rows affected (0.06 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [mattermost]> ALTER TABLE Users ROW_FORMAT=DYNAMIC;
Query OK, 25 rows affected (0.47 sec)              
Records: 25  Duplicates: 0  Warnings: 0

Topic		Replies	Views
Mattermost service not starting after upgrade Troubleshooting	2	754	April 25, 2024
Upgrading 5.31.1 Upgrade	1	349	July 19, 2023
[SOLVED] Upgrade Problem 3.6.2 to 3.8.2 Team Edition Troubleshooting	4	1508	May 12, 2017
After `sudo systemctl start mattermost`, there was no response Troubleshooting	1	515	October 21, 2021
Upgrade from 5.33.3 to 5.35.2 => Can't start mattermost Troubleshooting	7	859	July 8, 2021

Upgrade MM 5.3.1 to latest version fails

get MM version 5.9.8

backup current state

prepare new MM version

systemctl start mattermost

systemctl status mattermost

get MM version 5.9.8

backup current MM state

prepare new MM version

change MM installations

start and check “mattermost.service”

remove new version, restore old one

remove all database tables in db “mattermost”

restore previous database

start and check “mattermost.service”

remove old backups

mysql -u root -p

Related topics