Hello all,
since we updated our Mattermost version to 6.0, the service crashes several times a day.
System: CentOS Linux release 7.9.2009 (Core)
Mattermost version: 6.0.0 Team Edition
mmctl v6.0.0 – 85dd8887b1205800ba42128ddff58263c1d13925
Current version is 6.0.0 (6.0.0/Fri Oct 8 17:52:48 UTC 2021/c9b959e58a660fbc928dd91b062d7bcbb5657017/none)
Extract of /var/log/messages:
Oct 15 16:27:36 mattermost mattermost: panic: runtime error: invalid memory address or nil pointer dereference
Oct 15 16:27:36 mattermost mattermost: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xb85206]
Oct 15 16:27:36 mattermost mattermost: goroutine 6003 [running]:
Oct 15 16:27:36 mattermost mattermost: panic(0x24f0500, 0x3e90a50)
Oct 15 16:27:36 mattermost mattermost: runtime/panic.go:1065 +0x565 fp=0xc002b298c0 sp=0xc002b297f8 pc=0x439cc5
Oct 15 16:27:36 mattermost mattermost: runtime.panicmem()
Oct 15 16:27:36 mattermost mattermost: runtime/panic.go:212 +0x5b fp=0xc002b298e0 sp=0xc002b298c0 pc=0x437d1b
Oct 15 16:27:36 mattermost mattermost: runtime.sigpanic()
Oct 15 16:27:36 mattermost mattermost: runtime/signal_unix.go:734 +0x173 fp=0xc002b29918 sp=0xc002b298e0 pc=0x451a13
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/go-i18n/i18n/translation.(*template).Execute(0x0, 0x0, 0x0, 0x0, 0x7)
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/go-i18n@v1.11.0/i18n/translation/template.go:38 +0x26 fp=0xc002b29968 sp=0xc002b29918 pc=0xb85206
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/go-i18n/i18n/bundle.(*Bundle).translate(0xc000335050, 0xc004ef2138, 0x28fd183, 0x38, 0x0, 0x0, 0x0, 0x8, 0x10)
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/go-i18n@v1.11.0/i18n/bundle/bundle.go:395 +0x22a fp=0xc002b29a00 sp=0xc002b29968 pc=0xbe22ca
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/go-i18n/i18n/bundle.(*Bundle).TfuncAndLanguage.func1(0x28fd183, 0x38, 0x0, 0x0, 0x0, 0xb14ed8b4f44c120e, 0x10)
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/go-i18n@v1.11.0/i18n/bundle/bundle.go:310 +0x65 fp=0xc002b29a58 sp=0xc002b29a00 pc=0xbe2cc5
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/shared/i18n.tfuncWithFallback.func1(0x28fd183, 0x38, 0x0, 0x0, 0x0, 0xc0055cc680, 0x3d)
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/shared/i18n/i18n.go:139 +0x5e fp=0xc002b29ab8 sp=0xc002b29a58 pc=0xc657be
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/app/email.(*Service).sendBatchedEmailNotification(0xc002ec2500, 0xc005642d00, 0x1a, 0xc004dc2ab0, 0x1, 0x1)
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/app/email/email_batching.go:331 +0x1eab fp=0xc002b29f30 sp=0xc002b29ab8 pc=0x187e30b
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/app/email.(*Service).sendBatchedEmailNotification-fm(0xc005642d00, 0x1a, 0xc004dc2ab0, 0x1, 0x1)
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/app/email/email_batching.go:224 +0x65 fp=0xc002b29f70 sp=0xc002b29f30 pc=0x1880605
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/app/email.(*EmailBatchingJob).checkPendingNotifications.func1.1()
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/app/email/email_batching.go:204 +0x55 fp=0xc002b29fa8 sp=0xc002b29f70 pc=0x18801b5
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/app.(*Server).Go.func1(0xc0043ad6c0, 0xc00292a900)
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/app/server.go:1135 +0x27 fp=0xc002b29fd0 sp=0xc002b29fa8 pc=0x1f2a027
Oct 15 16:27:36 mattermost mattermost: runtime.goexit()
Oct 15 16:27:36 mattermost mattermost: runtime/asm_amd64.s:1371 +0x1 fp=0xc002b29fd8 sp=0xc002b29fd0 pc=0x474681
Oct 15 16:27:36 mattermost mattermost: created by github.com/mattermost/mattermost-server/v6/app.(*Server).Go
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/app/server.go:1134 +0x56
Oct 15 16:27:36 mattermost mattermost: goroutine 1 [chan receive, 16 minutes]:
Oct 15 16:27:36 mattermost mattermost: runtime.gopark(0x2961ef8, 0xc000200c58, 0x3f0170e, 0x2)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:336 +0xe5 fp=0xc0025a5ba8 sp=0xc0025a5b88 pc=0x43cd45
Oct 15 16:27:36 mattermost mattermost: runtime.chanrecv(0xc000200c00, 0x0, 0x3ed8601, 0x29622c8)
Oct 15 16:27:36 mattermost mattermost: runtime/chan.go:576 +0x36f fp=0xc0025a5c38 sp=0xc0025a5ba8 pc=0x40858f
Oct 15 16:27:36 mattermost mattermost: runtime.chanrecv1(0xc000200c00, 0x0)
Oct 15 16:27:36 mattermost mattermost: runtime/chan.go:439 +0x2b fp=0xc0025a5c68 sp=0xc0025a5c38 pc=0x4081cb
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/cmd/mattermost/commands.runServer(0xc0002c2770, 0xc000200c00, 0x0, 0x0)
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/cmd/mattermost/commands/server.go:119 +0x30b fp=0xc0025a5d48 sp=0xc0025a5c68 pc=0x228198b
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/cmd/mattermost/commands.serverCmdF(0x3ea5ea0, 0x3f0b068, 0x0, 0x0, 0x0, 0x0)
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/cmd/mattermost/commands/server.go:58 +0x165 fp=0xc0025a5db8 sp=0xc0025a5d48 pc=0x2281565
Oct 15 16:27:36 mattermost mattermost: github.com/spf13/cobra.(*Command).execute(0x3ea5ea0, 0xc00003c1f0, 0x0, 0x0, 0x3ea5ea0, 0xc00003c1f0)
Oct 15 16:27:36 mattermost mattermost: github.com/spf13/cobra@v1.1.3/command.go:852 +0x472 fp=0xc0025a5e78 sp=0xc0025a5db8 pc=0x2245852
Oct 15 16:27:36 mattermost mattermost: github.com/spf13/cobra.(*Command).ExecuteC(0x3ea5ea0, 0x0, 0xffffffff, 0xc000100058)
Oct 15 16:27:36 mattermost mattermost: github.com/spf13/cobra@v1.1.3/command.go:960 +0x375 fp=0xc0025a5f58 sp=0xc0025a5e78 pc=0x22463d5
Oct 15 16:27:36 mattermost mattermost: github.com/spf13/cobra.(*Command).Execute(...)
Oct 15 16:27:36 mattermost mattermost: github.com/spf13/cobra@v1.1.3/command.go:897
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/cmd/mattermost/commands.Run(...)
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/cmd/mattermost/commands/root.go:14
Oct 15 16:27:36 mattermost mattermost: main.main()
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/cmd/mattermost/main.go:31 +0x86 fp=0xc0025a5f88 sp=0xc0025a5f58 pc=0x22aca26
Oct 15 16:27:36 mattermost mattermost: runtime.main()
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:225 +0x256 fp=0xc0025a5fe0 sp=0xc0025a5f88 pc=0x43c916
Oct 15 16:27:36 mattermost mattermost: runtime.goexit()
Oct 15 16:27:36 mattermost mattermost: runtime/asm_amd64.s:1371 +0x1 fp=0xc0025a5fe8 sp=0xc0025a5fe0 pc=0x474681
Oct 15 16:27:36 mattermost mattermost: goroutine 2 [force gc (idle), 2 minutes]:
Oct 15 16:27:36 mattermost mattermost: runtime.gopark(0x2962108, 0x3ed4df0, 0x1411, 0x1)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:336 +0xe5 fp=0xc000058fb0 sp=0xc000058f90 pc=0x43cd45
Oct 15 16:27:36 mattermost mattermost: runtime.goparkunlock(...)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:342
Oct 15 16:27:36 mattermost mattermost: runtime.forcegchelper()
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:276 +0xc5 fp=0xc000058fe0 sp=0xc000058fb0 pc=0x43cba5
Oct 15 16:27:36 mattermost mattermost: runtime.goexit()
Oct 15 16:27:36 mattermost mattermost: runtime/asm_amd64.s:1371 +0x1 fp=0xc000058fe8 sp=0xc000058fe0 pc=0x474681
Oct 15 16:27:36 mattermost mattermost: created by runtime.init.6
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:264 +0x35
Oct 15 16:27:36 mattermost mattermost: goroutine 18 [GC sweep wait]:
Oct 15 16:27:36 mattermost mattermost: runtime.gopark(0x2962108, 0x3ed5200, 0x140c, 0x1)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:336 +0xe5 fp=0xc0000547a8 sp=0xc000054788 pc=0x43cd45
Oct 15 16:27:36 mattermost mattermost: runtime.goparkunlock(...)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:342
Oct 15 16:27:36 mattermost mattermost: runtime.bgsweep(0xc000096000)
Oct 15 16:27:36 mattermost mattermost: runtime/mgcsweep.go:182 +0x13b fp=0xc0000547d8 sp=0xc0000547a8 pc=0x426b5b
Oct 15 16:27:36 mattermost mattermost: runtime.goexit()
Oct 15 16:27:36 mattermost mattermost: runtime/asm_amd64.s:1371 +0x1 fp=0xc0000547e0 sp=0xc0000547d8 pc=0x474681
Oct 15 16:27:36 mattermost mattermost: created by runtime.gcenable
Oct 15 16:27:36 mattermost mattermost: runtime/mgc.go:217 +0x5c
Oct 15 16:27:36 mattermost mattermost: goroutine 19 [GC scavenge wait]:
Oct 15 16:27:36 mattermost mattermost: runtime.gopark(0x2962108, 0x3ed5ee0, 0x140d, 0x1)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:336 +0xe5 fp=0xc000054f78 sp=0xc000054f58 pc=0x43cd45
Oct 15 16:27:36 mattermost mattermost: runtime.goparkunlock(...)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:342
Oct 15 16:27:36 mattermost mattermost: runtime.bgscavenge(0xc000096000)
Oct 15 16:27:36 mattermost mattermost: runtime/mgcscavenge.go:314 +0x29c fp=0xc000054fd8 sp=0xc000054f78 pc=0x424cdc
Oct 15 16:27:36 mattermost mattermost: runtime.goexit()
Oct 15 16:27:36 mattermost mattermost: runtime/asm_amd64.s:1371 +0x1 fp=0xc000054fe0 sp=0xc000054fd8 pc=0x474681
Oct 15 16:27:36 mattermost mattermost: created by runtime.gcenable
Oct 15 16:27:36 mattermost mattermost: runtime/mgc.go:218 +0x7e
Oct 15 16:27:36 mattermost mattermost: goroutine 3 [finalizer wait, 14 minutes]:
Oct 15 16:27:36 mattermost mattermost: runtime.gopark(0x2962108, 0x3f0af78, 0xc005751410, 0x1)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:336 +0xe5 fp=0xc000058758 sp=0xc000058738 pc=0x43cd45
Oct 15 16:27:36 mattermost mattermost: runtime.goparkunlock(...)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:342
Oct 15 16:27:36 mattermost mattermost: runtime.runfinq()
Oct 15 16:27:36 mattermost mattermost: runtime/mfinal.go:175 +0xa9 fp=0xc0000587e0 sp=0xc000058758 pc=0x41bc69
Oct 15 16:27:36 mattermost mattermost: runtime.goexit()
Oct 15 16:27:36 mattermost mattermost: runtime/asm_amd64.s:1371 +0x1 fp=0xc0000587e8 sp=0xc0000587e0 pc=0x474681
Oct 15 16:27:36 mattermost mattermost: created by runtime.createfing
Oct 15 16:27:36 mattermost mattermost: runtime/mfinal.go:156 +0x65
Oct 15 16:27:36 mattermost mattermost: goroutine 4431 [select, 2 minutes]:
Oct 15 16:27:36 mattermost mattermost: runtime.gopark(0x2962160, 0x0, 0x1809, 0x1)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:336 +0xe5 fp=0xc002a3b828 sp=0xc002a3b808 pc=0x43cd45
Oct 15 16:27:36 mattermost mattermost: runtime.selectgo(0xc002a3bdb0, 0xc002a3ba30, 0x0, 0x0, 0xa, 0xc001b76201, 0x2, 0x1)
Oct 15 16:27:36 mattermost mattermost: runtime/select.go:327 +0xef7 fp=0xc002a3b960 sp=0xc002a3b828 pc=0x44e137
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/app.(*Hub).Start.func1()
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/app/web_hub.go:422 +0x47b fp=0xc002a3bfc0 sp=0xc002a3b960 pc=0x1f3159b
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/app.(*Hub).Start.func2()
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/app/web_hub.go:581 +0x4b fp=0xc002a3bfe0 sp=0xc002a3bfc0 pc=0x1f32cab
Oct 15 16:27:36 mattermost mattermost: runtime.goexit()
Oct 15 16:27:36 mattermost mattermost: runtime/asm_amd64.s:1371 +0x1 fp=0xc002a3bfe8 sp=0xc002a3bfe0 pc=0x474681
Oct 15 16:27:36 mattermost mattermost: created by github.com/mattermost/mattermost-server/v6/app.(*Hub).Start
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/app/web_hub.go:598 +0x10d
Oct 15 16:27:36 mattermost mattermost: goroutine 7 [GC worker (idle)]:
Oct 15 16:27:36 mattermost mattermost: runtime.gopark(0x2961fa0, 0xc0004f4280, 0x1418, 0x0)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:336 +0xe5 fp=0xc000059f60 sp=0xc000059f40 pc=0x43cd45
Oct 15 16:27:36 mattermost mattermost: runtime.gcBgMarkWorker()
Oct 15 16:27:36 mattermost mattermost: runtime/mgc.go:1911 +0xfb fp=0xc000059fe0 sp=0xc000059f60 pc=0x41f9bb
Oct 15 16:27:36 mattermost mattermost: runtime.goexit()
Oct 15 16:27:36 mattermost mattermost: runtime/asm_amd64.s:1371 +0x1 fp=0xc000059fe8 sp=0xc000059fe0 pc=0x474681
Oct 15 16:27:36 mattermost mattermost: created by runtime.gcBgMarkStartWorkers
Oct 15 16:27:36 mattermost mattermost: runtime/mgc.go:1835 +0x37
Oct 15 16:27:36 mattermost mattermost: goroutine 8 [GC worker (idle)]:
Oct 15 16:27:36 mattermost mattermost: runtime.gopark(0x2961fa0, 0xc0004f42a0, 0x1418, 0x0)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:336 +0xe5 fp=0xc00005a760 sp=0xc00005a740 pc=0x43cd45
Oct 15 16:27:36 mattermost mattermost: runtime.gcBgMarkWorker()
Oct 15 16:27:36 mattermost mattermost: runtime/mgc.go:1911 +0xfb fp=0xc00005a7e0 sp=0xc00005a760 pc=0x41f9bb
Oct 15 16:27:36 mattermost mattermost: runtime.goexit()
Oct 15 16:27:36 mattermost mattermost: runtime/asm_amd64.s:1371 +0x1 fp=0xc00005a7e8 sp=0xc00005a7e0 pc=0x474681
Oct 15 16:27:36 mattermost mattermost: created by runtime.gcBgMarkStartWorkers
Oct 15 16:27:36 mattermost mattermost: runtime/mgc.go:1835 +0x37
Oct 15 16:27:36 mattermost mattermost: goroutine 34 [GC worker (idle)]:
Oct 15 16:27:36 mattermost mattermost: runtime.gopark(0x2961fa0, 0xc000226000, 0x1418, 0x0)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:336 +0xe5 fp=0xc000222760 sp=0xc000222740 pc=0x43cd45
Oct 15 16:27:36 mattermost mattermost: runtime.gcBgMarkWorker()
Oct 15 16:27:36 mattermost mattermost: runtime/mgc.go:1911 +0xfb fp=0xc0002227e0 sp=0xc000222760 pc=0x41f9bb
Oct 15 16:27:36 mattermost mattermost: runtime.goexit()
Oct 15 16:27:36 mattermost mattermost: runtime/asm_amd64.s:1371 +0x1 fp=0xc0002227e8 sp=0xc0002227e0 pc=0x474681
Oct 15 16:27:36 mattermost mattermost: created by runtime.gcBgMarkStartWorkers
Oct 15 16:27:36 mattermost mattermost: runtime/mgc.go:1835 +0x37
Oct 15 16:27:36 mattermost mattermost: goroutine 35 [GC worker (idle)]:
Oct 15 16:27:36 mattermost mattermost: runtime.gopark(0x2961fa0, 0xc0004f42c0, 0x1418, 0x0)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:336 +0xe5 fp=0xc000222f60 sp=0xc000222f40 pc=0x43cd45
Oct 15 16:27:36 mattermost mattermost: runtime.gcBgMarkWorker()
Oct 15 16:27:36 mattermost mattermost: runtime/mgc.go:1911 +0xfb fp=0xc000222fe0 sp=0xc000222f60 pc=0x41f9bb
Oct 15 16:27:36 mattermost mattermost: runtime.goexit()
Oct 15 16:27:36 mattermost mattermost: runtime/asm_amd64.s:1371 +0x1 fp=0xc000222fe8 sp=0xc000222fe0 pc=0x474681
Oct 15 16:27:36 mattermost mattermost: created by runtime.gcBgMarkStartWorkers
Oct 15 16:27:36 mattermost mattermost: runtime/mgc.go:1835 +0x37
Oct 15 16:27:36 mattermost mattermost: goroutine 10 [select, 17 minutes]:
Oct 15 16:27:36 mattermost mattermost: runtime.gopark(0x2962160, 0x0, 0x1809, 0x1)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:336 +0xe5 fp=0xc00005ae08 sp=0xc00005ade8 pc=0x43cd45
Oct 15 16:27:36 mattermost mattermost: runtime.selectgo(0xc00005afa0, 0xc00005af80, 0x0, 0x0, 0x2, 0x1, 0x0, 0x0)
Oct 15 16:27:36 mattermost mattermost: runtime/select.go:327 +0xef7 fp=0xc00005af40 sp=0xc00005ae08 pc=0x44e137
Oct 15 16:27:36 mattermost mattermost: github.com/blevesearch/bleve/index.AnalysisWorker(0xc000bb29c0, 0xc000bb2a20)
Oct 15 16:27:36 mattermost mattermost: github.com/blevesearch/bleve@v1.0.14/index/analysis.go:102 +0xe5 fp=0xc00005afd0 sp=0xc00005af40 pc=0x1a2af85
Oct 15 16:27:36 mattermost mattermost: runtime.goexit()
Oct 15 16:27:36 mattermost mattermost: runtime/asm_amd64.s:1371 +0x1 fp=0xc00005afd8 sp=0xc00005afd0 pc=0x474681
Oct 15 16:27:36 mattermost mattermost: created by github.com/blevesearch/bleve/index.NewAnalysisQueue
Oct 15 16:27:36 mattermost mattermost: github.com/blevesearch/bleve@v1.0.14/index/analysis.go:94 +0xc8
Oct 15 16:27:36 mattermost mattermost: goroutine 11 [select, 17 minutes]:
Oct 15 16:27:36 mattermost mattermost: runtime.gopark(0x2962160, 0x0, 0x1809, 0x1)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:336 +0xe5 fp=0xc00005be08 sp=0xc00005bde8 pc=0x43cd45
Oct 15 16:27:36 mattermost mattermost: runtime.selectgo(0xc00005bfa0, 0xc00005bf80, 0x0, 0x0, 0x2, 0x1, 0x0, 0x0)
Oct 15 16:27:36 mattermost mattermost: runtime/select.go:327 +0xef7 fp=0xc00005bf40 sp=0xc00005be08 pc=0x44e137
Oct 15 16:27:36 mattermost mattermost: github.com/blevesearch/bleve/index.AnalysisWorker(0xc000bb29c0, 0xc000bb2a20)
Oct 15 16:27:36 mattermost mattermost: github.com/blevesearch/bleve@v1.0.14/index/analysis.go:102 +0xe5 fp=0xc00005bfd0 sp=0xc00005bf40 pc=0x1a2af85
Oct 15 16:27:36 mattermost mattermost: runtime.goexit()
Oct 15 16:27:36 mattermost mattermost: runtime/asm_amd64.s:1371 +0x1 fp=0xc00005bfd8 sp=0xc00005bfd0 pc=0x474681
Oct 15 16:27:36 mattermost mattermost: created by github.com/blevesearch/bleve/index.NewAnalysisQueue
Oct 15 16:27:36 mattermost mattermost: github.com/blevesearch/bleve@v1.0.14/index/analysis.go:94 +0xc8
Oct 15 16:27:36 mattermost mattermost: goroutine 12 [select, 17 minutes]:
Oct 15 16:27:36 mattermost mattermost: runtime.gopark(0x2962160, 0x0, 0x1809, 0x1)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:336 +0xe5 fp=0xc000055608 sp=0xc0000555e8 pc=0x43cd45
Oct 15 16:27:36 mattermost mattermost: runtime.selectgo(0xc0000557a0, 0xc000055780, 0x0, 0x0, 0x2, 0x1, 0x0, 0x0)
Oct 15 16:27:36 mattermost mattermost: runtime/select.go:327 +0xef7 fp=0xc000055740 sp=0xc000055608 pc=0x44e137
Oct 15 16:27:36 mattermost mattermost: github.com/blevesearch/bleve/index.AnalysisWorker(0xc000bb29c0, 0xc000bb2a20)
Oct 15 16:27:36 mattermost mattermost: github.com/blevesearch/bleve@v1.0.14/index/analysis.go:102 +0xe5 fp=0xc0000557d0 sp=0xc000055740 pc=0x1a2af85
Oct 15 16:27:36 mattermost mattermost: runtime.goexit()
Oct 15 16:27:36 mattermost mattermost: runtime/asm_amd64.s:1371 +0x1 fp=0xc0000557d8 sp=0xc0000557d0 pc=0x474681
Oct 15 16:27:36 mattermost mattermost: created by github.com/blevesearch/bleve/index.NewAnalysisQueue
Oct 15 16:27:36 mattermost mattermost: github.com/blevesearch/bleve@v1.0.14/index/analysis.go:94 +0xc8
Oct 15 16:27:36 mattermost mattermost: goroutine 13 [select, 17 minutes]:
Oct 15 16:27:36 mattermost mattermost: runtime.gopark(0x2962160, 0x0, 0x1809, 0x1)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:336 +0xe5 fp=0xc000055e08 sp=0xc000055de8 pc=0x43cd45
Oct 15 16:27:36 mattermost mattermost: runtime.selectgo(0xc000055fa0, 0xc000055f80, 0x0, 0x0, 0x2, 0x1, 0x0, 0x0)
Oct 15 16:27:36 mattermost mattermost: runtime/select.go:327 +0xef7 fp=0xc000055f40 sp=0xc000055e08 pc=0x44e137
Oct 15 16:27:36 mattermost mattermost: github.com/blevesearch/bleve/index.AnalysisWorker(0xc000bb29c0, 0xc000bb2a20)
Oct 15 16:27:36 mattermost mattermost: github.com/blevesearch/bleve@v1.0.14/index/analysis.go:102 +0xe5 fp=0xc000055fd0 sp=0xc000055f40 pc=0x1a2af85
Oct 15 16:27:36 mattermost mattermost: runtime.goexit()
Oct 15 16:27:36 mattermost mattermost: runtime/asm_amd64.s:1371 +0x1 fp=0xc000055fd8 sp=0xc000055fd0 pc=0x474681
Oct 15 16:27:36 mattermost mattermost: created by github.com/blevesearch/bleve/index.NewAnalysisQueue
Oct 15 16:27:36 mattermost mattermost: github.com/blevesearch/bleve@v1.0.14/index/analysis.go:94 +0xc8
Oct 15 16:27:36 mattermost mattermost: goroutine 4770 [select, 17 minutes]:
Oct 15 16:27:36 mattermost mattermost: runtime.gopark(0x2962160, 0x0, 0x1809, 0x1)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:336 +0xe5 fp=0xc000056520 sp=0xc000056500 pc=0x43cd45
Oct 15 16:27:36 mattermost mattermost: runtime.selectgo(0xc0000566f0, 0xc0000566c8, 0x0, 0x0, 0x2, 0x1, 0x1, 0x0)
Oct 15 16:27:36 mattermost mattermost: runtime/select.go:327 +0xef7 fp=0xc000056658 sp=0xc000056520 pc=0x44e137
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/jobs/active_users.(*Worker).Run(0xc001b6f640)
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/jobs/active_users/worker.go:61 +0x251 fp=0xc0000567d8 sp=0xc000056658 pc=0x22992d1
Oct 15 16:27:36 mattermost mattermost: runtime.goexit()
Oct 15 16:27:36 mattermost mattermost: runtime/asm_amd64.s:1371 +0x1 fp=0xc0000567e0 sp=0xc0000567d8 pc=0x474681
Oct 15 16:27:36 mattermost mattermost: created by github.com/mattermost/mattermost-server/v6/jobs.(*Workers).Start
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/jobs/workers.go:179 +0x425
Oct 15 16:27:36 mattermost mattermost: goroutine 4775 [select, 17 minutes]:
Oct 15 16:27:36 mattermost mattermost: runtime.gopark(0x2962160, 0x0, 0x1809, 0x1)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:336 +0xe5 fp=0xc000056d20 sp=0xc000056d00 pc=0x43cd45
Oct 15 16:27:36 mattermost mattermost: runtime.selectgo(0xc000056ef0, 0xc000056ec8, 0x0, 0x0, 0x2, 0x1, 0x1, 0x0)
Oct 15 16:27:36 mattermost mattermost: runtime/select.go:327 +0xef7 fp=0xc000056e58 sp=0xc000056d20 pc=0x44e137
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/jobs/export_delete.(*ExportDeleteWorker).Run(0xc001b6f780)
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/jobs/export_delete/worker.go:61 +0x251 fp=0xc000056fd8 sp=0xc000056e58 pc=0x22a51f1
Oct 15 16:27:36 mattermost mattermost: runtime.goexit()
Oct 15 16:27:36 mattermost mattermost: runtime/asm_amd64.s:1371 +0x1 fp=0xc000056fe0 sp=0xc000056fd8 pc=0x474681
Oct 15 16:27:36 mattermost mattermost: created by github.com/mattermost/mattermost-server/v6/jobs.(*Workers).Start
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/jobs/workers.go:199 +0x357
Oct 15 16:27:36 mattermost mattermost: goroutine 4856 [semacquire, 16 minutes]:
Oct 15 16:27:36 mattermost mattermost: runtime.gopark(0x2962108, 0x3ee2620, 0xc001061912, 0x4)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:336 +0xe5 fp=0xc0013d1630 sp=0xc0013d1610 pc=0x43cd45
Oct 15 16:27:36 mattermost mattermost: runtime.goparkunlock(...)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:342
Oct 15 16:27:36 mattermost mattermost: runtime.semacquire1(0xc001e4808c, 0x0, 0x1, 0x0)
Oct 15 16:27:36 mattermost mattermost: runtime/sema.go:144 +0x1a5 fp=0xc0013d1690 sp=0xc0013d1630 pc=0x44ec05
Oct 15 16:27:36 mattermost mattermost: sync.runtime_Semacquire(0xc001e4808c)
Oct 15 16:27:36 mattermost mattermost: runtime/sema.go:56 +0x45 fp=0xc0013d16c0 sp=0xc0013d1690 pc=0x470665
Oct 15 16:27:36 mattermost mattermost: sync.(*WaitGroup).Wait(0xc001e4808c)
Oct 15 16:27:36 mattermost mattermost: sync/waitgroup.go:130 +0x65 fp=0xc0013d16e8 sp=0xc0013d16c0 pc=0x481585
Oct 15 16:27:36 mattermost mattermost: github.com/hashicorp/go-plugin.(*Client).Start.func2(0xc001e48000, 0xc002ff8000)
Oct 15 16:27:36 mattermost mattermost: github.com/hashicorp/go-plugin@v1.4.2/client.go:627 +0xc5 fp=0xc0013d17d0 sp=0xc0013d16e8 pc=0x1183925
Oct 15 16:27:36 mattermost mattermost: runtime.goexit()
Oct 15 16:27:36 mattermost mattermost: runtime/asm_amd64.s:1371 +0x1 fp=0xc0013d17d8 sp=0xc0013d17d0 pc=0x474681
Oct 15 16:27:36 mattermost mattermost: created by github.com/hashicorp/go-plugin.(*Client).Start
Oct 15 16:27:36 mattermost mattermost: github.com/hashicorp/go-plugin@v1.4.2/client.go:614 +0xd37
Oct 15 16:27:36 mattermost mattermost: goroutine 4858 [chan receive, 16 minutes]:
Oct 15 16:27:36 mattermost mattermost: runtime.gopark(0x2961ef8, 0xc001b74e38, 0xc00141170e, 0x2)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:336 +0xe5 fp=0xc0002246c0 sp=0xc0002246a0 pc=0x43cd45
Oct 15 16:27:36 mattermost mattermost: runtime.chanrecv(0xc001b74de0, 0xc0002247a0, 0x1, 0x1)
Oct 15 16:27:36 mattermost mattermost: runtime/chan.go:576 +0x36f fp=0xc000224750 sp=0xc0002246c0 pc=0x40858f
Oct 15 16:27:36 mattermost mattermost: runtime.chanrecv2(0xc001b74de0, 0xc0002247a0, 0x0)
Oct 15 16:27:36 mattermost mattermost: runtime/chan.go:444 +0x2b fp=0xc000224780 sp=0xc000224750 pc=0x40820b
Oct 15 16:27:36 mattermost mattermost: github.com/hashicorp/go-plugin.(*Client).Start.func4.1(0xc001e48000, 0xc001b74de0)
Oct 15 16:27:36 mattermost mattermost: github.com/hashicorp/go-plugin@v1.4.2/client.go:673 +0x79 fp=0xc0002247d0 sp=0xc000224780 pc=0x1183f19
Oct 15 16:27:36 mattermost mattermost: runtime.goexit()
Oct 15 16:27:36 mattermost mattermost: runtime/asm_amd64.s:1371 +0x1 fp=0xc0002247d8 sp=0xc0002247d0 pc=0x474681
Oct 15 16:27:36 mattermost mattermost: created by github.com/hashicorp/go-plugin.(*Client).Start.func4
Oct 15 16:27:36 mattermost mattermost: github.com/hashicorp/go-plugin@v1.4.2/client.go:671 +0x49
Oct 15 16:27:36 mattermost mattermost: goroutine 4746 [select]:
Oct 15 16:27:36 mattermost mattermost: runtime.gopark(0x2962160, 0x0, 0x1809, 0x1)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:336 +0xe5 fp=0xc0002255b8 sp=0xc000225598 pc=0x43cd45
Oct 15 16:27:36 mattermost mattermost: runtime.selectgo(0xc000225768, 0xc00022573c, 0x0, 0x0, 0x2, 0x1, 0x1, 0x1)
Oct 15 16:27:36 mattermost mattermost: runtime/select.go:327 +0xef7 fp=0xc0002256f0 sp=0xc0002255b8 pc=0x44e137
Oct 15 16:27:36 mattermost mattermost: github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc002e41ec0, 0xc001415200, 0xc002f81620)
Oct 15 16:27:36 mattermost mattermost: github.com/go-sql-driver/mysql@v1.6.0/connection.go:614 +0xa5 fp=0xc0002257c8 sp=0xc0002256f0 pc=0xeb7b85
Oct 15 16:27:36 mattermost mattermost: runtime.goexit()
Oct 15 16:27:36 mattermost mattermost: runtime/asm_amd64.s:1371 +0x1 fp=0xc0002257d0 sp=0xc0002257c8 pc=0x474681
Oct 15 16:27:36 mattermost mattermost: created by github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher
Oct 15 16:27:36 mattermost mattermost: github.com/go-sql-driver/mysql@v1.6.0/connection.go:611 +0xbe
Oct 15 16:27:36 mattermost mattermost: goroutine 4857 [IO wait, 16 minutes]:
Oct 15 16:27:36 mattermost mattermost: runtime.gopark(0x29620e0, 0x7fafb42f9f70, 0xc000211b02, 0x5)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:336 +0xe5 fp=0xc0025a0c58 sp=0xc0025a0c38 pc=0x43cd45
Oct 15 16:27:36 mattermost mattermost: runtime.netpollblock(0x7fafb42f9f48, 0x72, 0xc002ffc026)
Oct 15 16:27:36 mattermost mattermost: runtime/netpoll.go:438 +0x9b fp=0xc0025a0c90 sp=0xc0025a0c58 pc=0x4352db
Oct 15 16:27:36 mattermost mattermost: internal/poll.runtime_pollWait(0x7fafb42f9f48, 0x72, 0xffffffffffffffff)
Oct 15 16:27:36 mattermost mattermost: runtime/netpoll.go:222 +0x55 fp=0xc0025a0cb8 sp=0xc0025a0c90 pc=0x46e635
Oct 15 16:27:36 mattermost mattermost: internal/poll.(*pollDesc).wait(0xc002fcff38, 0x72, 0xf01, 0xfda, 0xffffffffffffffff)
Oct 15 16:27:36 mattermost mattermost: internal/poll/fd_poll_runtime.go:87 +0x45 fp=0xc0025a0ce8 sp=0xc0025a0cb8 pc=0x4b28e5
Oct 15 16:27:36 mattermost mattermost: internal/poll.(*pollDesc).waitRead(...)
Oct 15 16:27:36 mattermost mattermost: internal/poll/fd_poll_runtime.go:92
Oct 15 16:27:36 mattermost mattermost: internal/poll.(*FD).Read(0xc002fcff20, 0xc002ffc026, 0xfda, 0xfda, 0x0, 0x0, 0x0)
Oct 15 16:27:36 mattermost mattermost: internal/poll/fd_unix.go:166 +0x1d5 fp=0xc0025a0d58 sp=0xc0025a0ce8 pc=0x4b39d5
Oct 15 16:27:36 mattermost mattermost: os.(*File).read(...)
Oct 15 16:27:36 mattermost mattermost: os/file_posix.go:31
Oct 15 16:27:36 mattermost mattermost: os.(*File).Read(0xc001b762c8, 0xc002ffc026, 0xfda, 0xfda, 0xc00248ac00, 0xc0025a0e88, 0x407985)
Oct 15 16:27:36 mattermost mattermost: os/file.go:117 +0x77 fp=0xc0025a0dd8 sp=0xc0025a0d58 pc=0x4bc977
Oct 15 16:27:36 mattermost mattermost: bufio.(*Scanner).Scan(0xc0025a0f10, 0xc0025a0f00)
Oct 15 16:27:36 mattermost mattermost: bufio/scan.go:214 +0xa9 fp=0xc0025a0ed0 sp=0xc0025a0dd8 pc=0x58fb69
Oct 15 16:27:36 mattermost mattermost: github.com/hashicorp/go-plugin.(*Client).Start.func3(0xc001e48000, 0xc001b74de0, 0x2cbc568, 0xc001b762c8)
Oct 15 16:27:36 mattermost mattermost: github.com/hashicorp/go-plugin@v1.4.2/client.go:660 +0x173 fp=0xc0025a0fc0 sp=0xc0025a0ed0 pc=0x1183e33
Oct 15 16:27:36 mattermost mattermost: runtime.goexit()
Oct 15 16:27:36 mattermost mattermost: runtime/asm_amd64.s:1371 +0x1 fp=0xc0025a0fc8 sp=0xc0025a0fc0 pc=0x474681
Oct 15 16:27:36 mattermost mattermost: created by github.com/hashicorp/go-plugin.(*Client).Start
Oct 15 16:27:36 mattermost mattermost: github.com/hashicorp/go-plugin@v1.4.2/client.go:655 +0xdc5
Oct 15 16:27:36 mattermost mattermost: goroutine 4750 [select, 17 minutes]:
Oct 15 16:27:36 mattermost mattermost: runtime.gopark(0x2962160, 0x0, 0x1809, 0x1)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:336 +0xe5 fp=0xc000f1f590 sp=0xc000f1f570 pc=0x43cd45
Oct 15 16:27:36 mattermost mattermost: runtime.selectgo(0xc000f1f768, 0xc000f1f734, 0x0, 0x0, 0x3, 0xc001ef9e01, 0xc000f1f730, 0x407415)
Oct 15 16:27:36 mattermost mattermost: runtime/select.go:327 +0xef7 fp=0xc000f1f6c8 sp=0xc000f1f590 pc=0x44e137
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/model.createTask.func1(0xc002d79dc0, 0x2260ff9290000, 0xc002e6f260)
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/model/scheduled_task.go:69 +0x230 fp=0xc000f1f7c8 sp=0xc000f1f6c8 pc=0xe6f5b0
Oct 15 16:27:36 mattermost mattermost: runtime.goexit()
Oct 15 16:27:36 mattermost mattermost: runtime/asm_amd64.s:1371 +0x1 fp=0xc000f1f7d0 sp=0xc000f1f7c8 pc=0x474681
Oct 15 16:27:36 mattermost mattermost: created by github.com/mattermost/mattermost-server/v6/model.createTask
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/model/scheduled_task.go:46 +0x105
Oct 15 16:27:36 mattermost mattermost: goroutine 4655 [select]:
Oct 15 16:27:36 mattermost mattermost: runtime.gopark(0x2962160, 0x0, 0x1809, 0x1)
Oct 15 16:27:36 mattermost mattermost: runtime/proc.go:336 +0xe5 fp=0xc002a3fd90 sp=0xc002a3fd70 pc=0x43cd45
Oct 15 16:27:36 mattermost mattermost: runtime.selectgo(0xc002a3ff68, 0xc002a3ff34, 0x0, 0x0, 0x3, 0xc002e77101, 0x1, 0x407401)
Oct 15 16:27:36 mattermost mattermost: runtime/select.go:327 +0xef7 fp=0xc002a3fec8 sp=0xc002a3fd90 pc=0x44e137
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/model.createTask.func1(0xc001b6f200, 0x6fc23ac00, 0xc002ec15e0)
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/model/scheduled_task.go:69 +0x230 fp=0xc002a3ffc8 sp=0xc002a3fec8 pc=0xe6f5b0
Oct 15 16:27:36 mattermost mattermost: runtime.goexit()
Oct 15 16:27:36 mattermost mattermost: runtime/asm_amd64.s:1371 +0x1 fp=0xc002a3ffd0 sp=0xc002a3ffc8 pc=0x474681
Oct 15 16:27:36 mattermost mattermost: created by github.com/mattermost/mattermost-server/v6/model.createTask
Oct 15 16:27:36 mattermost systemd: mattermost.service: main process exited, code=killed, status=6/ABRT
Oct 15 16:27:36 mattermost mattermost: github.com/mattermost/mattermost-server/v6/model/scheduled_task.go:46 +0x105
Oct 15 16:27:36 mattermost mattermost: goroutine 4735 [select, 17 minutes]:
....
Any ideas how we can solve this problem?
Thank you!