Upgrade from 5.33.3 to 5.35.2 => Can't start mattermost

DMT · June 4, 2021, 3:25pm

Summary
After following the upgrade guide (Upgrading Mattermost Server — Mattermost 5.35 documentation) trying to update from 5.33.3 to 5.35.2 I cannot start the Mattermost service.

Steps to reproduce
Start the mattermost service (sudo systemctl start mattermost.service).

Expected behavior
I would expect that the mattermost service starts normally.

Observed behavior
Here are the logs:
journalctl -xe yields

Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]:         runtime/chan.go:577 +0x36f fp=0xc001572ea0 sp=0xc001572e10 pc=0x4084ef
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: runtime.chanrecv2(0xc0029cb680, 0xc001572f68, 0xc000201901)
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]:         runtime/chan.go:444 +0x2b fp=0xc001572ed0 sp=0xc001572ea0 pc=0x40816b
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: github.com/getsentry/sentry-go.(*HTTPTransport).worker(0xc002978c00)
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]:         github.com/getsentry/sentry-go@v0.10.0/transport.go:378 +0x7d fp=0xc001572fd8 sp=0xc001572ed0 pc=0x19dd7dd
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: runtime.goexit()
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]:         runtime/asm_amd64.s:1374 +0x1 fp=0xc001572fe0 sp=0xc001572fd8 pc=0x472ca1
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: created by github.com/getsentry/sentry-go.(*HTTPTransport).Configure.func1
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]:         github.com/getsentry/sentry-go@v0.10.0/transport.go:249 +0x3e
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: goroutine 3743 [select]:
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: runtime.gopark(0x2c7adb0, 0x0, 0x1809, 0x1)
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]:         runtime/proc.go:306 +0xe5 fp=0xc002b31828 sp=0xc002b31808 pc=0x43c705
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: runtime.selectgo(0xc002b31d20, 0xc002b31a10, 0x8, 0x1, 0x1)
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]:         runtime/select.go:338 +0xcef fp=0xc002b31950 sp=0xc002b31828 pc=0x44ca4f
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: github.com/mattermost/mattermost-server/v5/app.(*Hub).Start.func1()
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]:         github.com/mattermost/mattermost-server/v5/app/web_hub.go:393 +0x413 fp=0xc002b31fc0 sp=0xc002b31950 pc=0x20a5433
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: github.com/mattermost/mattermost-server/v5/app.(*Hub).Start.func2()
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]:         github.com/mattermost/mattermost-server/v5/app/web_hub.go:503 +0x4b fp=0xc002b31fe0 sp=0xc002b31fc0 pc=0x20a66cb
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: runtime.goexit()
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]:         runtime/asm_amd64.s:1374 +0x1 fp=0xc002b31fe8 sp=0xc002b31fe0 pc=0x472ca1
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: created by github.com/mattermost/mattermost-server/v5/app.(*Hub).Start
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]:         github.com/mattermost/mattermost-server/v5/app/web_hub.go:520 +0x10d
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: goroutine 3859 [select]:
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: runtime.gopark(0x2c7adb0, 0x0, 0x1809, 0x1)
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]:         runtime/proc.go:306 +0xe5 fp=0xc0012e4530 sp=0xc0012e4510 pc=0x43c705
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: runtime.selectgo(0xc0012e4700, 0xc0012e468c, 0x3, 0xc00004c800, 0xc0012e46f8)
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]:         runtime/select.go:338 +0xcef fp=0xc0012e4658 sp=0xc0012e4530 pc=0x44ca4f
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: github.com/mattermost/mattermost-server/v5/shared/templates.NewWithWatcher.func1(0xc000c6b6e0, 0xc0029bf600, 0xc000095a40, 0xc001c92e80, 0x19)
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]:         github.com/mattermost/mattermost-server/v5/shared/templates/templates.go:88 +0x1b9 fp=0xc0012e47b8 sp=0xc0012e4658 pc=0x124a3f9
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: runtime.goexit()
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]:         runtime/asm_amd64.s:1374 +0x1 fp=0xc0012e47c0 sp=0xc0012e47b8 pc=0x472ca1
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: created by github.com/mattermost/mattermost-server/v5/shared/templates.NewWithWatcher
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]:         github.com/mattermost/mattermost-server/v5/shared/templates/templates.go:82 +0x235
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: goroutine 3731 [select]:
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: runtime.gopark(0x2c7adb0, 0x0, 0x1809, 0x1)
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]:         runtime/proc.go:306 +0xe5 fp=0xc001dc7828 sp=0xc001dc7808 pc=0x43c705
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: runtime.selectgo(0xc001dc7d20, 0xc001dc7a10, 0x8, 0x1, 0x1)
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]:         runtime/select.go:338 +0xcef fp=0xc001dc7950 sp=0xc001dc7828 pc=0x44ca4f
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: github.com/mattermost/mattermost-server/v5/app.(*Hub).Start.func1()
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]:         github.com/mattermost/mattermost-server/v5/app/web_hub.go:393 +0x413 fp=0xc001dc7fc0 sp=0xc001dc7950 pc=0x20a5433
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: github.com/mattermost/mattermost-server/v5/app.(*Hub).Start.func2()
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]:         github.com/mattermost/mattermost-server/v5/app/web_hub.go:503 +0x4b fp=0xc001dc7fe0 sp=0xc001dc7fc0 pc=0x20a66cb
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: runtime.goexit()
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]:         runtime/asm_amd64.s:1374 +0x1 fp=0xc001dc7fe8 sp=0xc001dc7fe0 pc=0x472ca1
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]: created by github.com/mattermost/mattermost-server/v5/app.(*Hub).Start
Jun 04 16:55:36 ldap.diamontech.local mattermost[13773]:         github.com/mattermost/mattermost-server/v5/app/web_hub.go:520 +0x10d
Jun 04 16:55:36 ldap.diamontech.local systemd[1]: mattermost.service: Main process exited, code=dumped, status=6/ABRT
Jun 04 16:55:36 ldap.diamontech.local systemd[1]: mattermost.service: Failed with result 'core-dump'.
Jun 04 16:55:36 ldap.diamontech.local systemd[1]: Failed to start Mattermost.
-- Subject: Unit mattermost.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit mattermost.service has failed.
--
-- The result is RESULT.

# systemctl status mattermost.service yields:

systemctl status mattermost.service
● mattermost.service - Mattermost
   Loaded: loaded (/lib/systemd/system/mattermost.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: signal) since Fri 2021-06-04 16:55:26 CEST; 9s ago
  Process: 13464 ExecStart=/opt/mattermost/bin/mattermost (code=killed, signal=ABRT)
 Main PID: 13464 (code=killed, signal=ABRT)

The mattermost.log file got overwritten as I’ve reverted the installation to 5.33.3 but reviewing it at 5.35.2 didn’t show any errors, just continuous infos that MySQL database is being pinged so no info from there.

From the special installations notes page I didn’t see anything I would have needed to do in order to update to 5.35.2 but maybe I’ve missed some configuration or something I needed to do prior to update from 5.33.3?

Any ideas or should I wait for the next update release? Thanks for help

amy.blais · June 4, 2021, 4:06pm

@streamer45 / @agnivade ^

streamer45 · June 4, 2021, 4:38pm

@DMT

Are you able to share the full journalctl log until right before the crash happened? The one provided looks cut. Also, what MySQL version are you running?

DMT · June 4, 2021, 5:24pm

@streamer45
Thanks for the answer!
MySQL:

mysql  Ver 14.14 Distrib 5.7.34, for Linux (x86_64) using  EditLine wrapper

journalctl: I’ve exported the file from the first attempt to start with new version through lots of restarts and troubleshooting starting at aroun 16:06 and ending with a successfull start after downgrade back to 5.33.3 at around 17:00.I can’t make sense of this log to be honst.
I’ve uploaded the file here: journal_trunc.txt - Google Drive as it was too big to be posted here.

Thanks for the help!
Any idea?

streamer45 · June 4, 2021, 5:33pm

Thank you @DTM. I think that log is enough info to fix this issue as I was able to reproduce.

justvanbloom · June 7, 2021, 2:54am

Same issue, but from 5.32

DMT · July 8, 2021, 3:36pm

Is this solved with updating to 5.36.1?

amy.blais · July 8, 2021, 10:49pm

It was fixed in v5.35.3 dot release, so 5.36.1 also has the fix.

Topic		Replies	Views
After `sudo systemctl start mattermost`, there was no response Troubleshooting	1	506	October 21, 2021
Unable to upgrade past v5.33.5 Troubleshooting	0	448	August 31, 2021
Mattermost service not starting after upgrade Troubleshooting	2	419	April 25, 2024
Upgrading 5.31.1 Upgrade	1	321	July 19, 2023
Upgrade MM 5.3.1 to latest version fails Troubleshooting	74	5065	February 23, 2021

Upgrade from 5.33.3 to 5.35.2 => Can't start mattermost

Related topics