FocalBoards errors and drop down a service

Troubleshooting
1

Good day! Today we catch a some list of errors in Focalboard, which drops our MM down and fixed only by reboot of all service.
Can you help us with this problem/ Or, maybe, i didn’t find the same issue in Jira, which does goes earlier.

Summary

Errors in Mattermost and focalboards, which drop down the service and caused a forced reboot to return it back on.
Link to original post: Issue Link
Code of some Errors

error [2023-06-16 09:49:18.615 +05:00] GetPreferencesForUser: We encountered an error while finding preferences.

error [2023-06-16 09:49:18.148 +05:00] access denied to team

error [2023-06-16 09:49:18.155 +05:00] API ERROR code=500 error=EOF api=/api/v2/teams

error [2023-06-16 09:49:18.568 +05:00] read tcp XXXXXXXXXXXXXXX: read: connection reset by peer

2

Hi @Beast-Breacher and welcome to the Mattermost forums!

What mattermost server version is this and what deployment method (binary, omnibus, gitlab-integrated, docker, etc.) did you choose? Or is this the focalboard personal server?

3

Hi @agriesser
We use a 7.9-2 server version and docker deployment method.

4

Alright, this could be a problem with a specific user or a database migration issue.
Can you please provide the docker logs output for the application container after you restarted it? I’d like to see all logs during the server start.
You can also find them in the mattermost.log file, if you have access to it.

5

Ok. We will send some info after the end of my colleague’s vacation. On next week.

6

@agriesser Good Day. We have receive the logs from Mattermost after restart. How can i send it to you?

7

{“timestamp”:“2023-07-10 23:41:04.822 +05:00”,“level”:“info”,“msg”:“Stopping Server…”,“caller”:“app/server.go:666”}
{“timestamp”:“2023-07-10 23:41:04.851 +05:00”,“level”:“info”,“msg”:“Stopped workers”,“caller”:“jobs/workers.go:90”}
{“timestamp”:“2023-07-10 23:41:04.851 +05:00”,“level”:“info”,“msg”:“Stopping schedulers.”,“caller”:“jobs/schedulers.go:118”}
{“timestamp”:“2023-07-10 23:41:04.851 +05:00”,“level”:“info”,“msg”:“Schedulers stopped.”,“caller”:“jobs/schedulers.go:50”}
{“timestamp”:“2023-07-10 23:41:04.851 +05:00”,“level”:“info”,“msg”:“Shutting down plugins”,“caller”:“app/plugin.go:379”}
{“timestamp”:“2023-07-10 23:41:04.855 +05:00”,“level”:“info”,“msg”:“plugin process exited”,“caller”:“plugin/hclog_adapter.go:61”,“plugin_id”:“com.mattermost.plugin-staff-profiles”,“wrapped_extras”:“path/usr/local/src/mattermost/plugins/com.mattermost.plugin-staff-profiles/server/dist/plugin-linux-amd64pid3300147”}
{“timestamp”:“2023-07-10 23:41:04.863 +05:00”,“level”:“info”,“msg”:“plugin process exited”,“caller”:“plugin/hclog_adapter.go:61”,“plugin_id”:“com.mattermost.plugin-readonly-channel”,“wrapped_extras”:“path/usr/local/src/mattermost/plugins/com.mattermost.plugin-readonly-channel/server/dist/plugin-linux-amd64pid2843912”}
{“timestamp”:“2023-07-10 23:41:04.865 +05:00”,“level”:“info”,“msg”:“plugin process exited”,“caller”:“plugin/hclog_adapter.go:61”,“plugin_id”:“com.mattermost.plugin-kontur-staff”,“wrapped_extras”:“path/usr/local/src/mattermost/plugins/com.mattermost.plugin-kontur-staff/server/dist/plugin-linux-amd64pid2843921”}
{“timestamp”:“2023-07-10 23:41:04.894 +05:00”,“level”:“info”,“msg”:“plugin process exited”,“caller”:“plugin/hclog_adapter.go:61”,“plugin_id”:“ktalk”,“wrapped_extras”:“path/usr/local/src/mattermost/plugins/ktalk/server/dist/plugin-linux-amd64pid2843893”}
{“timestamp”:“2023-07-10 23:41:04.894 +05:00”,“level”:“info”,“msg”:“plugin process exited”,“caller”:“plugin/hclog_adapter.go:61”,“plugin_id”:“telephone.webrtc.mattermost.plugin.fix”,“wrapped_extras”:“path/usr/local/src/mattermost/plugins/telephone.webrtc.mattermost.plugin.fix/server/dist/plugin-linux-amd64pid2843904”}
{“timestamp”:“2023-07-10 23:41:04.897 +05:00”,“level”:“info”,“msg”:“Server.Shutdown”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“focalboard”}
{“timestamp”:“2023-07-10 23:41:04.902 +05:00”,“level”:“info”,“msg”:“plugin process exited”,“caller”:“plugin/hclog_adapter.go:61”,“plugin_id”:“com.mattermost.apps”,“wrapped_extras”:“path/usr/local/src/mattermost/plugins/com.mattermost.apps/server/dist/plugin-linux-amd64pid2843954”}
{“timestamp”:“2023-07-10 23:41:04.952 +05:00”,“level”:“info”,“msg”:“plugin process exited”,“caller”:“plugin/hclog_adapter.go:61”,“plugin_id”:“focalboard”,“wrapped_extras”:“path/usr/local/src/mattermost/plugins/focalboard/server/dist/plugin-linux-amd64pid2843939”}
{“timestamp”:“2023-07-10 23:41:04.952 +05:00”,“level”:“info”,“msg”:“stopping websocket hub connections”,“caller”:“platform/web_hub.go:115”}
{“timestamp”:“2023-07-10 23:42:36.329 +05:00”,“level”:“info”,“msg”:“Server is initializing…”,“caller”:“platform/service.go:165”,“go_version”:“go1.18.1”}
{“timestamp”:“2023-07-10 23:42:36.329 +05:00”,“level”:“info”,“msg”:“Pinging SQL”,“caller”:“sqlstore/store.go:242”,“database”:“master”,“dataSource”:“postgres://%2A%2A%2A%2A:%2A%2A%2A%2A@hasql-pg.kontur:6432/mattermost?binary_parameters=yes&connect_timeout=10&sslmode=disable”}
{“timestamp”:“2023-07-10 23:42:36.492 +05:00”,“level”:“info”,“msg”:“Starting websocket hubs”,“caller”:“platform/web_hub.go:95”,“number_of_hubs”:16}
{“timestamp”:“2023-07-10 23:42:36.523 +05:00”,“level”:“info”,“msg”:“Current version is 7.9.3 (7.9-2/Wed Apr 19 12:55:08 UTC 2023/116a9f7efd752389dccebf8e967af30701ec323e/none)”,“caller”:“app/server.go:409”,“current_version”:“7.9.3”,“build_number”:“7.9-2”,“build_date”:“Wed Apr 19 12:55:08 UTC 2023”,“build_hash”:“116a9f7efd752389dccebf8e967af30701ec323e”,“build_hash_enterprise”:“none”}
{“timestamp”:“2023-07-10 23:42:36.523 +05:00”,“level”:“info”,“msg”:“Team Edition Build”,“caller”:“app/server.go:420”,“enterprise_build”:false}
{“timestamp”:“2023-07-10 23:42:36.523 +05:00”,“level”:“info”,“msg”:“Printing current working”,“caller”:“app/server.go:424”,“directory”:“/opt/mattermost”}
{“timestamp”:“2023-07-10 23:42:36.523 +05:00”,“level”:“info”,“msg”:“Loaded config”,“caller”:“app/server.go:425”,“source”:“file:///opt/mattermost/config/config.json”}
{“timestamp”:“2023-07-10 23:42:36.533 +05:00”,“level”:“info”,“msg”:“Starting workers”,“caller”:“jobs/workers.go:48”}
{“timestamp”:“2023-07-10 23:42:36.533 +05:00”,“level”:“info”,“msg”:“Starting schedulers.”,“caller”:“jobs/schedulers.go:47”}
{“timestamp”:“2023-07-10 23:42:36.782 +05:00”,“level”:“info”,“msg”:“Starting up plugins”,“caller”:“app/plugin.go:218”}
{“timestamp”:“2023-07-10 23:42:36.782 +05:00”,“level”:“info”,“msg”:“Syncing plugins from the file store”,“caller”:“app/plugin.go:299”}
{“timestamp”:“2023-07-10 23:42:36.804 +05:00”,“level”:“info”,“msg”:“Syncing plugin from file store”,“caller”:“app/plugin.go:359”,“bundle”:“plugins/ktalk.tar.gz”}
{“timestamp”:“2023-07-10 23:42:36.804 +05:00”,“level”:“info”,“msg”:“Syncing plugin from file store”,“caller”:“app/plugin.go:359”,“bundle”:“plugins/com.mattermost.plugin-kontur-staff.tar.gz”}
{“timestamp”:“2023-07-10 23:42:36.804 +05:00”,“level”:“info”,“msg”:“Syncing plugin from file store”,“caller”:“app/plugin.go:359”,“bundle”:“plugins/com.mattermost.plugin-readonly-channel.tar.gz”}
{“timestamp”:“2023-07-10 23:42:36.805 +05:00”,“level”:“info”,“msg”:“Syncing plugin from file store”,“caller”:“app/plugin.go:359”,“bundle”:“plugins/com.mattermost.plugin-staff-profiles.tar.gz”}
{“timestamp”:“2023-07-10 23:42:36.805 +05:00”,“level”:“info”,“msg”:“Syncing plugin from file store”,“caller”:“app/plugin.go:359”,“bundle”:“plugins/telephone.webrtc.mattermost.plugin.fix.tar.gz”}
{“timestamp”:“2023-07-10 23:42:36.805 +05:00”,“level”:“info”,“msg”:“Syncing plugin from file store”,“caller”:“app/plugin.go:359”,“bundle”:“plugins/telephone.webrtc.mattermost.plugin.tar.gz”}
{“timestamp”:“2023-07-10 23:42:36.809 +05:00”,“level”:“info”,“msg”:“Syncing plugin from file store”,“caller”:“app/plugin.go:359”,“bundle”:“plugins/com.mattermost.plugin-telephone-template-new.tar.gz”}
{“timestamp”:“2023-07-10 23:42:36.810 +05:00”,“level”:“info”,“msg”:“Syncing plugin from file store”,“caller”:“app/plugin.go:359”,“bundle”:“plugins/focalboard.tar.gz”}
{“timestamp”:“2023-07-10 23:42:37.883 +05:00”,“level”:“warn”,“msg”:“Failed to perform initial product notices fetch”,“caller”:“app/server.go:451”,“error”:“UpdateProductNotices: Fetching product notices failed, Get "https://notices.mattermost.com/\”: dial tcp 52.85.158.16:443: connect: connection refused"}
{“timestamp”:“2023-07-10 23:42:42.799 +05:00”,“level”:“info”,“msg”:“connectDatabase”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“focalboard”,“dbType”:“postgres”}
{“timestamp”:“2023-07-10 23:42:42.842 +05:00”,“level”:“info”,“msg”:“Schema migration table is correct format”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“focalboard”}
{“timestamp”:“2023-07-10 23:42:42.843 +05:00”,“level”:“info”,“msg”:“{"level":"info","msg":"Pinging SQL","fields":{"database":"master"}}\n”,“caller”:“io/io.go:428”,“plugin_id”:“focalboard”,“source”:“plugin_stderr”}
{“timestamp”:“2023-07-10 23:42:42.920 +05:00”,“level”:“info”,“msg”:“Initialized notification backend”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“focalboard”,“name”:“notifyMentions”}
{“timestamp”:“2023-07-10 23:42:42.920 +05:00”,“level”:“info”,“msg”:“Initialized notification backend”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“focalboard”,“name”:“notifySubscriptions”}
{“timestamp”:“2023-07-10 23:42:42.921 +05:00”,“level”:“info”,“msg”:“Initialized notification backend”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“focalboard”,“name”:“notifyLogger”}
{“timestamp”:“2023-07-10 23:42:42.938 +05:00”,“level”:“info”,“msg”:“Focalboard server”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“focalboard”,“version”:“7.9.3”,“edition”:“plugin”,“build_number”:“4565904155”,“build_date”:“"Thu Mar 30 15:05:51 UTC 2023"”,“build_hash”:“c9e8dd3d8d0355783e4aa39f6a0caef4a253b7db”}
{“timestamp”:“2023-07-10 23:42:42.938 +05:00”,“level”:“info”,“msg”:“Server.Start”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“focalboard”}
{“timestamp”:“2023-07-10 23:42:42.938 +05:00”,“level”:“info”,“msg”:“Boards product successfully started.”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“focalboard”}
{“timestamp”:“2023-07-10 23:42:44.554 +05:00”,“level”:“info”,“msg”:“activated”,“caller”:“app/plugin_api.go:973”,“plugin_id”:“com.mattermost.apps”,“plugin_caller”:“server/plugin.go:135”}
{“timestamp”:“2023-07-10 23:42:44.939 +05:00”,“level”:“info”,“msg”:“Starting Server…”,“caller”:“app/server.go:892”}
{“timestamp”:“2023-07-10 23:42:44.939 +05:00”,“level”:“info”,“msg”:“Server is listening on [::]:8065”,“caller”:“app/server.go:964”,“address”:“[::]:8065”}
{“timestamp”:“2023-07-10 23:42:44.939 +05:00”,“level”:“info”,“msg”:“Sending systemd READY notification.”,“caller”:“commands/server.go:130”}
{“timestamp”:“2023-07-10 23:42:55.603 +05:00”,“level”:“info”,“msg”:“Post.Message has size restrictions”,“caller”:“sqlstore/post_store.go:2544”,“max_characters”:16383,“max_bytes”:65535}
{“timestamp”:“2023-07-10 23:43:21.059 +05:00”,“level”:“info”,“msg”:“Draft.Message has size restrictions”,“caller”:“sqlstore/draft_store.go:257”,“max_characters”:16383,“max_bytes”:65535}
{“timestamp”:“2023-07-10 23:45:21.156 +05:00”,“level”:“error”,“msg”:“access denied to team”,“caller”:“app/plugin_api.go:976”,“plugin_id”:“focalboard”}
{“timestamp”:“2023-07-10 23:53:44.223 +05:00”,“level”:“info”,“msg”:“SimpleWorker: Job is complete”,“caller”:“jobs/base_workers.go:96”,“worker”:“ExpiryNotify”,“job_id”:“c1x6ykyt878aue7t9zphkhfhwc”}
{“timestamp”:“2023-07-11 00:04:44.406 +05:00”,“level”:“info”,“msg”:“SimpleWorker: Job is complete”,“caller”:“jobs/base_workers.go:96”,“worker”:“ExpiryNotify”,“job_id”:“pfra7oyzh3nyzrw3aieauyb1wa”}
{“timestamp”:“2023-07-11 00:15:44.493 +05:00”,“level”:“info”,“msg”:“SimpleWorker: Job is complete”,“caller”:“jobs/base_workers.go:96”,“worker”:“ExpiryNotify”,“job_id”:“nd7rx7etctyourecqq861fdyiw”}
{“timestamp”:“2023-07-11 00:26:44.560 +05:00”,“level”:“info”,“msg”:“SimpleWorker: Job is complete”,“caller”:“jobs/base_workers.go:96”,“worker”:“ExpiryNotify”,“job_id”:“ab9etznpfibbb8j7bj4nypmrfo”}
{“timestamp”:“2023-07-11 00:37:44.686 +05:00”,“level”:“info”,“msg”:“SimpleWorker: Job is complete”,“caller”:“jobs/base_workers.go:96”,“worker”:“ExpiryNotify”,“job_id”:“hwkg64ur4fy8znbsswo3dhsy7a”}
{“timestamp”:“2023-07-11 00:42:46.072 +05:00”,“level”:“error”,“msg”:“Worker: Failed to fetch product notices”,“caller”:"product_notices/worker.go

8

@agriesser Good Day! Do you receive our logs?

9

@Beast-Breacher I have the same problem. What was the reason? Did you manage to solve it?