[Solved] Server startup fails: "panic: interface conversion"

Summary
Server stopped working and now will not start

Steps to reproduce

  1. Ubuntu 18.04
  2. Was running v7.7.0 when it failed
  3. I updated to v7.10.0
  4. Starting server with sudo systemctl start mattermost.service fails. See logs below.

Expected behavior
Expected server to start normally.

Observed behavior

$ sudo systemctl start mattermost.service
Job for mattermost.service failed because the control process exited with error code.
See "systemctl status mattermost.service" and "journalctl -xe" for details.

$ systemctl status mattermost.service
‚óŹ mattermost.service - Mattermost
   Loaded: loaded (/lib/systemd/system/mattermost.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Tue 2023-05-09 09:37:51 PDT; 1s ago
  Process: 4000 ExecStart=/opt/mattermost/bin/mattermost (code=exited, status=2)
 Main PID: 4000 (code=exited, status=2)

$ sudo journalctl -xe
-- Unit mattermost.service has finished shutting down.
May 09 09:39:16 myserver systemd[1]: Starting Mattermost...
-- Subject: Unit mattermost.service has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit mattermost.service has begun starting up.
May 09 09:39:16 myserver mattermost[4232]: panic: interface conversion: interface {} is nil, not bool
May 09 09:39:16 myserver mattermost[4232]: goroutine 1 [running]:
May 09 09:39:16 myserver mattermost[4232]: github.com/mattermost/mattermost-server/v6/model.(*ProductSettings).SetDefaults(...)
May 09 09:39:16 myserver mattermost[4232]:         github.com/mattermost/mattermost-server/v6/model/config.go:2808
May 09 09:39:16 myserver mattermost[4232]: github.com/mattermost/mattermost-server/v6/model.(*Config).SetDefaults(0xc000cf0600)
May 09 09:39:16 myserver mattermost[4232]:         github.com/mattermost/mattermost-server/v6/model/config.go:3304 +0x17ae
May 09 09:39:16 myserver mattermost[4232]: github.com/mattermost/mattermost-server/v6/config.(*Store).Load(0xc000498310)
May 09 09:39:16 myserver mattermost[4232]:         github.com/mattermost/mattermost-server/v6/config/store.go:285 +0x265
May 09 09:39:16 myserver mattermost[4232]: github.com/mattermost/mattermost-server/v6/config.NewStoreFromBacking({0x353b820?, 0xc00250d060}, 0x0, 0x0)
May 09 09:39:16 myserver mattermost[4232]:         github.com/mattermost/mattermost-server/v6/config/store.go:79 +0x95
May 09 09:39:16 myserver mattermost[4232]: github.com/mattermost/mattermost-server/v6/config.NewStoreFromDSN({0x2db48be, 0xb}, 0x0?, 0x0?, 0x0?)
May 09 09:39:16 myserver mattermost[4232]:         github.com/mattermost/mattermost-server/v6/config/store.go:100 +0x11a
May 09 09:39:16 myserver mattermost[4232]: github.com/mattermost/mattermost-server/v6/cmd/mattermost/commands.serverCmdF(0x5342820?, {0x2d9640f?, 0x0?, 0x0?})
May 09 09:39:16 myserver mattermost[4232]:         github.com/mattermost/mattermost-server/v6/cmd/mattermost/commands/server.go:52 +0xb8
May 09 09:39:16 myserver mattermost[4232]: github.com/spf13/cobra.(*Command).execute(0x5342820, {0xc000052210, 0x0, 0x0})
May 09 09:39:16 myserver mattermost[4232]:         github.com/spf13/cobra@v1.6.1/command.go:916 +0x862
May 09 09:39:16 myserver mattermost[4232]: github.com/spf13/cobra.(*Command).ExecuteC(0x5342820)
May 09 09:39:16 myserver mattermost[4232]:         github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bd
May 09 09:39:16 myserver mattermost[4232]: github.com/spf13/cobra.(*Command).Execute(...)
May 09 09:39:16 myserver mattermost[4232]:         github.com/spf13/cobra@v1.6.1/command.go:968
May 09 09:39:16 myserver mattermost[4232]: github.com/mattermost/mattermost-server/v6/cmd/mattermost/commands.Run(...)
May 09 09:39:16 myserver mattermost[4232]:         github.com/mattermost/mattermost-server/v6/cmd/mattermost/commands/root.go:14
May 09 09:39:16 myserver mattermost[4232]: main.main()
May 09 09:39:16 myserver mattermost[4232]:         github.com/mattermost/mattermost-server/v6/cmd/mattermost/main.go:20 +0x87
May 09 09:39:16 myserver systemd[1]: mattermost.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 09 09:39:16 myserver systemd[1]: mattermost.service: Failed with result 'exit-code'.
May 09 09:39:16 myserver systemd[1]: Failed to start Mattermost.
-- Subject: Unit mattermost.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit mattermost.service has failed.
--
-- The result is RESULT.
May 09 09:39:20 myserver sudo[4260]:    username : TTY=pts/0 ; PWD=/mnt/volume_tor1_01 ; USER=root ; COMMAND=/bin/journalctl -xe
May 09 09:39:20 myserver sudo[4260]: pam_unix(sudo:session): session opened for user root by username(uid=0)
lines 1351-1393/1393 (END)

Here are the last 100 lines from mattermost.log.

{"timestamp":"2023-05-08 22:03:20.679 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"qcfm54ski3nsfmp613d8o84nre","error":"read tcp 127.0.0.1:8065->127.0.0.1:54850: i/o timeout"}
{"timestamp":"2023-05-08 22:07:20.082 -07:00","level":"info","msg":"Worker: Job is complete","caller":"scheduler/worker.go:89","worker":"Plugins","job_id":"xbjae5x9s7dw78nswiu4wx3fue"}
{"timestamp":"2023-05-08 22:07:20.082 -07:00","level":"info","msg":"SimpleWorker: Job is complete","caller":"jobs/base_workers.go:88","worker":"ExportDelete","job_id":"3pionn516jgebb5r61zszmhzzw"}
{"timestamp":"2023-05-08 22:07:20.082 -07:00","level":"info","msg":"SimpleWorker: Job is complete","caller":"jobs/base_workers.go:88","worker":"ImportDelete","job_id":"iuy8x48gctbk7m1mmp3zbauhco"}
{"timestamp":"2023-05-08 22:07:20.109 -07:00","level":"info","msg":"SimpleWorker: Job is complete","caller":"jobs/base_workers.go:88","worker":"ProductNotices","job_id":"re6u9dkdojbxj8jpu19x3yi9kh"}
{"timestamp":"2023-05-08 22:12:25.904 -07:00","level":"info","msg":"api endpoint requires a license","caller":"web/context.go:115","path":"/api/v4/groups","request_id":"6o916metnp8i3cta8tcsi3tozc","ip_addr":"174.7.246.117","user_id":"dpjk5oiwr38n7n4bgdk6n6fa9a","method":"GET","err_where":"","http_code":501,"error":": api endpoint requires a license"}
{"timestamp":"2023-05-08 22:12:26.985 -07:00","level":"info","msg":"api endpoint requires a license","caller":"web/context.go:115","path":"/api/v4/users/dpjk5oiwr38n7n4bgdk6n6fa9a/groups","request_id":"9qwezj4t4fn5581pghmwodskcr","ip_addr":"174.7.246.117","user_id":"dpjk5oiwr38n7n4bgdk6n6fa9a","method":"GET","err_where":"","http_code":501,"error":": api endpoint requires a license"}
{"timestamp":"2023-05-08 22:12:27.328 -07:00","level":"info","msg":"api endpoint requires a license","caller":"web/context.go:115","path":"/api/v4/groups","request_id":"rrzk78uacpnqiescuhia47167e","ip_addr":"174.7.246.117","user_id":"dpjk5oiwr38n7n4bgdk6n6fa9a","method":"GET","err_where":"","http_code":501,"error":": api endpoint requires a license"}
{"timestamp":"2023-05-08 22:13:48.379 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"95pppegiojnttbbgu5wu1x7b9o","error":"read tcp 127.0.0.1:8065->127.0.0.1:48374: i/o timeout"}
{"timestamp":"2023-05-08 22:20:11.014 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:49564: i/o timeout"}
{"timestamp":"2023-05-08 22:27:49.808 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:38986: i/o timeout"}
{"timestamp":"2023-05-08 22:36:40.780 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"95pppegiojnttbbgu5wu1x7b9o","error":"read tcp 127.0.0.1:8065->127.0.0.1:47980: i/o timeout"}
{"timestamp":"2023-05-08 22:44:42.118 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:43110: i/o timeout"}
{"timestamp":"2023-05-08 22:47:06.825 -07:00","level":"info","msg":"api endpoint requires a license","caller":"web/context.go:115","path":"/api/v4/users/8boi3nw9ctyztr1bnybescn59h/groups","request_id":"ta3e5a9dupfj98ym9edftz4jey","ip_addr":"162.156.184.211","user_id":"8boi3nw9ctyztr1bnybescn59h","method":"GET","err_where":"","http_code":501,"error":": api endpoint requires a license"}
{"timestamp":"2023-05-08 22:58:06.925 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"95pppegiojnttbbgu5wu1x7b9o","error":"read tcp 127.0.0.1:8065->127.0.0.1:36702: i/o timeout"}
{"timestamp":"2023-05-08 23:02:08.799 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:57144: i/o timeout"}
{"timestamp":"2023-05-08 23:07:21.656 -07:00","level":"info","msg":"SimpleWorker: Job is complete","caller":"jobs/base_workers.go:88","worker":"ProductNotices","job_id":"qfe3wmx1ubbfdeuz5ise1xbo8r"}
{"timestamp":"2023-05-08 23:15:56.674 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"95pppegiojnttbbgu5wu1x7b9o","error":"read tcp 127.0.0.1:8065->127.0.0.1:54386: i/o timeout"}
{"timestamp":"2023-05-08 23:18:04.353 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:46038: i/o timeout"}
{"timestamp":"2023-05-08 23:20:19.313 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:46030: i/o timeout"}
{"timestamp":"2023-05-08 23:22:35.328 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:37790: i/o timeout"}
{"timestamp":"2023-05-08 23:35:44.581 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"95pppegiojnttbbgu5wu1x7b9o","error":"read tcp 127.0.0.1:8065->127.0.0.1:35958: i/o timeout"}
{"timestamp":"2023-05-08 23:53:53.145 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"95pppegiojnttbbgu5wu1x7b9o","error":"read tcp 127.0.0.1:8065->127.0.0.1:53804: i/o timeout"}
{"timestamp":"2023-05-09 00:06:51.628 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"5t1b1n1pjtgyxnxiwrpjcob6oe","error":"read tcp 127.0.0.1:8065->127.0.0.1:53762: i/o timeout"}
{"timestamp":"2023-05-09 00:07:08.150 -07:00","level":"info","msg":"SimpleWorker: Job is complete","caller":"jobs/base_workers.go:88","worker":"ProductNotices","job_id":"u5k9x4rm5b83ifdr11x8at7fse"}
{"timestamp":"2023-05-09 00:14:19.138 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"95pppegiojnttbbgu5wu1x7b9o","error":"read tcp 127.0.0.1:8065->127.0.0.1:33380: i/o timeout"}
{"timestamp":"2023-05-09 00:19:48.438 -07:00","level":"info","msg":"api endpoint requires a license","caller":"web/context.go:115","path":"/api/v4/users/n5x7it9zi7ytmjffrgx95azgho/groups","request_id":"u4kh915hi3bj9fornm3o43p99r","ip_addr":"162.156.184.211","user_id":"n5x7it9zi7ytmjffrgx95azgho","method":"GET","err_where":"","http_code":501,"error":": api endpoint requires a license"}
{"timestamp":"2023-05-09 00:26:10.628 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:34522: i/o timeout"}
{"timestamp":"2023-05-09 00:31:04.216 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"95pppegiojnttbbgu5wu1x7b9o","error":"read tcp 127.0.0.1:8065->127.0.0.1:45562: i/o timeout"}
{"timestamp":"2023-05-09 00:37:09.131 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"5t1b1n1pjtgyxnxiwrpjcob6oe","error":"read tcp 127.0.0.1:8065->127.0.0.1:40928: i/o timeout"}
{"timestamp":"2023-05-09 00:43:23.430 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:57190: i/o timeout"}
{"timestamp":"2023-05-09 00:55:26.162 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"5t1b1n1pjtgyxnxiwrpjcob6oe","error":"read tcp 127.0.0.1:8065->127.0.0.1:50918: i/o timeout"}
{"timestamp":"2023-05-09 00:56:17.041 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:50900: i/o timeout"}
{"timestamp":"2023-05-09 01:07:10.016 -07:00","level":"info","msg":"SimpleWorker: Job is complete","caller":"jobs/base_workers.go:88","worker":"ProductNotices","job_id":"gfim8xbq1jfabgen5mpg1ptwro"}
{"timestamp":"2023-05-09 01:13:33.604 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:33468: i/o timeout"}
{"timestamp":"2023-05-09 01:31:21.907 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:54978: i/o timeout"}
{"timestamp":"2023-05-09 01:38:07.457 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"95pppegiojnttbbgu5wu1x7b9o","error":"read tcp 127.0.0.1:8065->127.0.0.1:55494: i/o timeout"}
{"timestamp":"2023-05-09 01:42:40.233 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"qcfm54ski3nsfmp613d8o84nre","error":"read tcp 127.0.0.1:8065->127.0.0.1:35168: i/o timeout"}
{"timestamp":"2023-05-09 01:48:31.396 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:47734: i/o timeout"}
{"timestamp":"2023-05-09 02:00:13.850 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"qcfm54ski3nsfmp613d8o84nre","error":"read tcp 127.0.0.1:8065->127.0.0.1:55062: i/o timeout"}
{"timestamp":"2023-05-09 02:04:30.098 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:42894: i/o timeout"}
{"timestamp":"2023-05-09 02:07:11.353 -07:00","level":"info","msg":"SimpleWorker: Job is complete","caller":"jobs/base_workers.go:88","worker":"ProductNotices","job_id":"eyj9i8zig7bp38ygj419mmatoy"}
{"timestamp":"2023-05-09 02:21:31.424 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:40610: i/o timeout"}
{"timestamp":"2023-05-09 02:35:53.969 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"qcfm54ski3nsfmp613d8o84nre","error":"read tcp 127.0.0.1:8065->127.0.0.1:56548: i/o timeout"}
{"timestamp":"2023-05-09 02:39:14.679 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:41442: i/o timeout"}
{"timestamp":"2023-05-09 02:56:09.904 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"5t1b1n1pjtgyxnxiwrpjcob6oe","error":"read tcp 127.0.0.1:8065->127.0.0.1:51284: i/o timeout"}
{"timestamp":"2023-05-09 03:01:07.496 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"95pppegiojnttbbgu5wu1x7b9o","error":"read tcp 127.0.0.1:8065->127.0.0.1:48826: i/o timeout"}
{"timestamp":"2023-05-09 03:07:13.179 -07:00","level":"info","msg":"SimpleWorker: Job is complete","caller":"jobs/base_workers.go:88","worker":"ProductNotices","job_id":"ma6jheb8rfb5ff88ijx7m68ruo"}
{"timestamp":"2023-05-09 03:31:40.721 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"5t1b1n1pjtgyxnxiwrpjcob6oe","error":"read tcp 127.0.0.1:8065->127.0.0.1:33178: i/o timeout"}
{"timestamp":"2023-05-09 03:55:48.623 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:45612: i/o timeout"}
{"timestamp":"2023-05-09 03:58:50.417 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"5t1b1n1pjtgyxnxiwrpjcob6oe","error":"read tcp 127.0.0.1:8065->127.0.0.1:46356: i/o timeout"}
{"timestamp":"2023-05-09 04:06:54.520 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"qcfm54ski3nsfmp613d8o84nre","error":"read tcp 127.0.0.1:8065->127.0.0.1:50658: i/o timeout"}
{"timestamp":"2023-05-09 04:07:14.760 -07:00","level":"info","msg":"SimpleWorker: Job is complete","caller":"jobs/base_workers.go:88","worker":"ProductNotices","job_id":"duf3cx9wstfpjfbts6drn7ar5e"}
{"timestamp":"2023-05-09 04:08:07.432 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"95pppegiojnttbbgu5wu1x7b9o","error":"read tcp 127.0.0.1:8065->127.0.0.1:42800: i/o timeout"}
{"timestamp":"2023-05-09 04:09:41.484 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:44576: i/o timeout"}
{"timestamp":"2023-05-09 04:16:11.695 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"5t1b1n1pjtgyxnxiwrpjcob6oe","error":"read tcp 127.0.0.1:8065->127.0.0.1:35022: i/o timeout"}
{"timestamp":"2023-05-09 04:26:59.929 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:60430: i/o timeout"}
{"timestamp":"2023-05-09 04:44:11.635 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:52886: i/o timeout"}
{"timestamp":"2023-05-09 05:02:46.144 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:41250: i/o timeout"}
{"timestamp":"2023-05-09 05:07:05.322 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"95pppegiojnttbbgu5wu1x7b9o","error":"read tcp 127.0.0.1:8065->127.0.0.1:56020: i/o timeout"}
{"timestamp":"2023-05-09 05:07:15.928 -07:00","level":"info","msg":"SimpleWorker: Job is complete","caller":"jobs/base_workers.go:88","worker":"ProductNotices","job_id":"3fjbchrrgfnofcgt7kgj1qfr3h"}
{"timestamp":"2023-05-09 05:20:45.622 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:38504: i/o timeout"}
{"timestamp":"2023-05-09 05:36:57.293 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:50894: i/o timeout"}
{"timestamp":"2023-05-09 05:46:46.537 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"qcfm54ski3nsfmp613d8o84nre","error":"read tcp 127.0.0.1:8065->127.0.0.1:55182: i/o timeout"}
{"timestamp":"2023-05-09 05:49:12.020 -07:00","level":"info","msg":"api endpoint requires a license","caller":"web/context.go:115","path":"/api/v4/users/e3enefdn53ftimcck185114mdc/groups","request_id":"bc3qauh9ibyz9pacgx5zs7c5zo","ip_addr":"162.156.184.211","user_id":"e3enefdn53ftimcck185114mdc","method":"GET","err_where":"","http_code":501,"error":": api endpoint requires a license"}
{"timestamp":"2023-05-09 05:53:22.371 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:36568: i/o timeout"}
{"timestamp":"2023-05-09 06:07:17.295 -07:00","level":"info","msg":"SimpleWorker: Job is complete","caller":"jobs/base_workers.go:88","worker":"ProductNotices","job_id":"7uwx1nq5yf86feii967b6f58sa"}
{"timestamp":"2023-05-09 06:11:48.458 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:60088: i/o timeout"}
{"timestamp":"2023-05-09 06:16:55.414 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"5t1b1n1pjtgyxnxiwrpjcob6oe","error":"read tcp 127.0.0.1:8065->127.0.0.1:60458: i/o timeout"}
{"timestamp":"2023-05-09 06:24:02.357 -07:00","level":"info","msg":"api endpoint requires a license","caller":"web/context.go:115","path":"/api/v4/users/pzrdxi7hjjb7fpfynoahgam9ca/groups","request_id":"jput4nmmttgtzcrmq59botxjyh","ip_addr":"135.23.126.167","user_id":"pzrdxi7hjjb7fpfynoahgam9ca","method":"GET","err_where":"","http_code":501,"error":": api endpoint requires a license"}
{"timestamp":"2023-05-09 06:29:08.686 -07:00","level":"error","msg":"websocket.NextReader: closing websocket","caller":"app/web_conn.go:830","user_id":"8boi3nw9ctyztr1bnybescn59h","error":"read tcp 127.0.0.1:8065->127.0.0.1:60428: i/o timeout"}
{"timestamp":"2023-05-09 06:29:45.530 -07:00","level":"info","msg":"Stopping Server...","caller":"app/server.go:872"}
{"timestamp":"2023-05-09 06:29:45.530 -07:00","level":"info","msg":"stopping websocket hub connections","caller":"app/web_hub.go:113"}
{"timestamp":"2023-05-09 06:29:45.535 -07:00","level":"error","msg":"RPC call OnWebSocketDisconnect to plugin failed.","caller":"plugin/client_rpc_generated.go:620","plugin_id":"com.mattermost.calls","error":"write unix @->/tmp/plugin3534050682: write: broken pipe"}
{"timestamp":"2023-05-09 06:29:45.543 -07:00","level":"error","msg":"plugin process exited","caller":"plugin/hclog_adapter.go:79","plugin_id":"com.mattermost.calls","wrapped_extras":"pathplugins/com.mattermost.calls/server/dist/plugin-linux-amd64pid1175errorsignal: terminated"}
{"timestamp":"2023-05-09 06:29:45.544 -07:00","level":"error","msg":"plugin process exited","caller":"plugin/hclog_adapter.go:79","plugin_id":"com.github.moussetc.mattermost.plugin.spoiler","wrapped_extras":"pathplugins/com.github.moussetc.mattermost.plugin.spoiler/server/dist/plugin-linux-amd64pid1148errorsignal: terminated"}
{"timestamp":"2023-05-09 06:29:45.545 -07:00","level":"error","msg":"plugin process exited","caller":"plugin/hclog_adapter.go:79","plugin_id":"com.github.matterpoll.matterpoll","wrapped_extras":"pathplugins/com.github.matterpoll.matterpoll/server/dist/plugin-linux-amd64pid1120errorsignal: terminated"}
{"timestamp":"2023-05-09 06:29:45.547 -07:00","level":"error","msg":"RPC call OnWebSocketDisconnect to plugin failed.","caller":"plugin/client_rpc_generated.go:620","plugin_id":"com.mattermost.calls","error":"connection is shut down"}
{"timestamp":"2023-05-09 06:29:45.562 -07:00","level":"error","msg":"RPC call OnWebSocketDisconnect to plugin failed.","caller":"plugin/client_rpc_generated.go:620","plugin_id":"com.mattermost.calls","error":"connection is shut down"}
{"timestamp":"2023-05-09 06:29:45.566 -07:00","level":"error","msg":"RPC call OnWebSocketDisconnect to plugin failed.","caller":"plugin/client_rpc_generated.go:620","plugin_id":"com.mattermost.calls","error":"connection is shut down"}
{"timestamp":"2023-05-09 06:29:45.571 -07:00","level":"error","msg":"RPC call OnWebSocketDisconnect to plugin failed.","caller":"plugin/client_rpc_generated.go:620","plugin_id":"com.mattermost.calls","error":"connection is shut down"}
{"timestamp":"2023-05-09 06:29:45.575 -07:00","level":"error","msg":"RPC call OnWebSocketDisconnect to plugin failed.","caller":"plugin/client_rpc_generated.go:620","plugin_id":"com.mattermost.calls","error":"connection is shut down"}
{"timestamp":"2023-05-09 06:29:45.579 -07:00","level":"error","msg":"RPC call OnWebSocketDisconnect to plugin failed.","caller":"plugin/client_rpc_generated.go:620","plugin_id":"com.mattermost.calls","error":"connection is shut down"}
{"timestamp":"2023-05-09 06:29:45.585 -07:00","level":"error","msg":"RPC call OnWebSocketDisconnect to plugin failed.","caller":"plugin/client_rpc_generated.go:620","plugin_id":"com.mattermost.calls","error":"connection is shut down"}
{"timestamp":"2023-05-09 06:29:45.588 -07:00","level":"error","msg":"RPC call OnWebSocketDisconnect to plugin failed.","caller":"plugin/client_rpc_generated.go:620","plugin_id":"com.mattermost.calls","error":"connection is shut down"}
{"timestamp":"2023-05-09 06:29:45.594 -07:00","level":"error","msg":"RPC call OnWebSocketDisconnect to plugin failed.","caller":"plugin/client_rpc_generated.go:620","plugin_id":"com.mattermost.calls","error":"connection is shut down"}
{"timestamp":"2023-05-09 06:29:45.598 -07:00","level":"error","msg":"RPC call OnWebSocketDisconnect to plugin failed.","caller":"plugin/client_rpc_generated.go:620","plugin_id":"com.mattermost.calls","error":"connection is shut down"}
{"timestamp":"2023-05-09 06:29:45.617 -07:00","level":"info","msg":"Stopped workers","caller":"jobs/workers.go:90"}
{"timestamp":"2023-05-09 06:29:45.617 -07:00","level":"info","msg":"Stopping schedulers.","caller":"jobs/schedulers.go:118"}
{"timestamp":"2023-05-09 06:29:45.617 -07:00","level":"info","msg":"Schedulers stopped.","caller":"jobs/schedulers.go:50"}
{"timestamp":"2023-05-09 06:29:45.617 -07:00","level":"info","msg":"Shutting down plugins","caller":"app/plugin.go:366"}
{"timestamp":"2023-05-09 06:29:45.618 -07:00","level":"error","msg":"RPC call OnDeactivate to plugin failed.","caller":"plugin/client_rpc_generated.go:33","plugin_id":"com.github.matterpoll.matterpoll","error":"connection is shut down"}
{"timestamp":"2023-05-09 06:29:45.618 -07:00","level":"warn","msg":"error closing client during Kill","caller":"plugin/hclog_adapter.go:70","plugin_id":"com.github.matterpoll.matterpoll","wrapped_extras":"errconnection is shut down"}
{"timestamp":"2023-05-09 06:29:45.618 -07:00","level":"warn","msg":"plugin failed to exit gracefully","caller":"plugin/hclog_adapter.go:72","plugin_id":"com.github.matterpoll.matterpoll"}
{"timestamp":"2023-05-09 06:29:45.618 -07:00","level":"error","msg":"RPC call OnDeactivate to plugin failed.","caller":"plugin/client_rpc_generated.go:33","plugin_id":"com.mattermost.calls","error":"connection is shut down"}
{"timestamp":"2023-05-09 06:29:45.618 -07:00","level":"warn","msg":"error closing client during Kill","caller":"plugin/hclog_adapter.go:70","plugin_id":"com.mattermost.calls","wrapped_extras":"errconnection is shut down"}
{"timestamp":"2023-05-09 06:29:45.618 -07:00","level":"warn","msg":"plugin failed to exit gracefully","caller":"plugin/hclog_adapter.go:72","plugin_id":"com.mattermost.calls"}
{"timestamp":"2023-05-09 06:29:45.618 -07:00","level":"warn","msg":"error closing client during Kill","caller":"plugin/hclog_adapter.go:70","plugin_id":"com.github.moussetc.mattermost.plugin.spoiler","wrapped_extras":"errconnection is shut down"}
{"timestamp":"2023-05-09 06:29:45.618 -07:00","level":"warn","msg":"plugin failed to exit gracefully","caller":"plugin/hclog_adapter.go:72","plugin_id":"com.github.moussetc.mattermost.plugin.spoiler"}
{"timestamp":"2023-05-09 06:29:45.620 -07:00","level":"info","msg":"Server stopped","caller":"app/server.go:965"}

I never figured out the problem but solved it by reinstalling Mattermost from scratch and restoring from a backup.