6.7.2 -> 7.10.5 migration error messages

Troubleshooting
1

Upgraded self-hosted teams version of mattermost from 6.7.2 to 7.10.5. Seeing error/warn messages such as following in mattermost.log:

"error","msg":"Not activating plugin because diagnostics are disabled","caller":"app/plugin_api.go:976","plugin_id":"com.mattermost.nps"}
"error","msg":"Failed to install prepackaged plugin","caller":"app/plugin.go:964","path":"/opt/mattermost/prepackaged_plugins/mattermost-plugin-nps-v1.3.1-linux-amd64.tar.gz","error":"Failed to install extracted prepackaged plugin /opt/mattermost/prepackaged_plugins/mattermost-plugin-nps-v1.3.1-linux-amd64.tar.gz: installExtractedPlugin: Unable to restart plugin on upgrade., Not activating plugin because diagnostics are disabled"}
"warn","msg":"Rudder credentials are not set. Disabling analytics.","caller":"app/plugin_api.go:979","plugin_id":"playbooks","plugin_caller":"github.com/mattermost/mattermost-plugin-playbooks/server/plugin.go:141"}
"warn","msg":"found collation/charset mismatch, fixing table","caller":"app/plugin_api.go:979","plugin_id":"focalboard","tableName":"focalboard_schema_migrations","tableCollation":"utf8mb3_general_ci","tableCharSet":"utf8mb3","collation":"utf8mb4_general_ci","charSet":"utf8mb4"}
"error","msg":"Table creation / migration failed","caller":"app/plugin_api.go:976","plugin_id":"focalboard","error":"\"error running fix collations and charsets migration: error fetching charSet: sql: no rows in result set\""}
"error","msg":"Failed to install prepackaged plugin","caller":"app/plugin.go:964","path":"/opt/mattermost/prepackaged_plugins/focalboard-v7.10.5-linux-amd64.tar.gz","error":"Failed to install extracted prepackaged plugin /opt/mattermost/prepackaged_plugins/focalboard-v7.10.5-linux-amd64.tar.gz: installExtractedPlugin: Unable to restart plugin on upgrade., cannot activate plugin: error initializing the DB: error running fix collations and charsets migration: error fetching charSet: sql: no rows in result set"}
"error","msg":"Not activating plugin because diagnostics are disabled","caller":"app/plugin_api.go:976","plugin_id":"com.mattermost.nps"}
"error","msg":"Unable to activate plugin","caller":"app/plugin.go:171","plugin_id":"com.mattermost.nps","error":"Not activating plugin because diagnostics are disabled"}
"warn","msg":"Your push notification server is configured with HTTP. For improved security, update to HTTPS in your configuration.","caller":"app/server.go:1178"}

Full post-upgrade log excerpt in this gist

Are these warns/errors something to worry about? Note mattermost chat seems to work fine, seeing no errors. Also note this installation has never created any boards.

Edit: forgot to mention I’m using MariaDB v15.1

2

I don’t see anything I’d lose sleep over, so long as you aren’t stopping at 7.10.5! It’s been EOL for a year and a half. Please see Release policy - Mattermost documentation

You really should have gone to 7.1.9, then 7.8.15, then 8.1.13, then 9.5.12, then 9.11.4 At that point you could go to 10.x or wait for 10.5 which will be our next ESR. See Version archive - Mattermost documentation for older versions.

Upgrade steps are at Upgrade Mattermost Server - Mattermost documentation

1 Like
3

Thanks for the reply. No I’m not stopping there. So far did 6.3.9 → 6.7.2 (i.e. latest v6) → 7.10.5 (i.e. latest v7). From your message I understand one should upgrade from ESR->ESR, without skipping ESR versions in a given major ver? (ie. upgrading both to 7.1.9 AND 7.8.15, not skipping 7.1.9?).

That was my understanding as well, but got a bit bamboozled by the Available only on Enterprise plans tag on release policy page and assumed for Team releases that’s no longer the case/supported.

As for the errors, what is worrying to me a bit is the collation/charset mismatch message followed by Table creation / migration failed. I’m guessing automatic collation change failed and I need to manually fix them. Running the db (MariaDB v15.1) query to see collations, there are mostly utf8mb4, but a few are utf8mb3:

MariaDB [mattermost]> SELECT CHARACTER_SET_NAME from information_schema.columns where table_schema=(SELECT DATABASE()) and CHARACTER_SET_NAME is not null;
+--------------------+
| CHARACTER_SET_NAME |
+--------------------+
| utf8mb4            |
| utf8mb3            |
/-/
+--------------------+
505 rows in set (0.006 sec)

Full query result in this gist

And what are those errors regarding plugins? Failed to install prepackaged plugin, Unable to activate plugin and such?

4

Correct, upgrades should go ESR → ESR

I have a feeling that making a big jump that skipped ESR may have wound up corrupting or copying an incompatible plugin.

I think you can look in /opt/mattermost/prepackaged_plugins and see if there are old ones, permissions issues that prevented an upgrade, etc. They should be deleted and replaced during an upgrade.

Did you start with the Teams (free) version? You may have oerwritten that with the licensable version, which would also cause issues. If you start with Teams, you need to make sure you’re following the upgrade with newer Teams versions.

6

Don’t know, all plugins have been updated with latest restart, so am guessing they’re coming pre-packaged with the latest mmost version:

$ ls -lt /mattermost/plugins/
total 0
drwxr--r--    1 root     root           103 Oct 30 00:46 com.mattermost.calls
drwxr--r--    1 root     root           111 Oct 30 00:46 focalboard
drwxr--r--    1 root     root            99 Oct 30 00:46 playbooks
drwxr--r--    1 root     root            71 Oct 30 00:46 com.mattermost.apps
drwxr--r--    1 root     root            85 Oct 30 00:46 com.mattermost.nps

$  ls -lt /opt/mattermost/prepackaged_plugins/
total 276204
-rw-r--r--    1 root     root      20987184 Jul 24  2023 mattermost-plugin-apps-v1.2.1-linux-amd64.tar.gz
-rw-r--r--    1 root     root           566 Jul 24  2023 mattermost-plugin-apps-v1.2.1-linux-amd64.tar.gz.sig
-rw-r--r--    1 root     root      46184002 Jul 24  2023 focalboard-v7.10.5-linux-amd64.tar.gz
-rw-r--r--    1 root     root           566 Jul 24  2023 focalboard-v7.10.5-linux-amd64.tar.gz.sig
-rw-r--r--    1 root     root      12393224 Jul 24  2023 mattermost-plugin-welcomebot-v1.3.0-linux-amd64.tar.gz
-rw-r--r--    1 root     root           566 Jul 24  2023 mattermost-plugin-welcomebot-v1.3.0-linux-amd64.tar.gz.sig
-rw-r--r--    1 root     root      10775971 Jul 24  2023 mattermost-plugin-zoom-v1.6.0-linux-amd64.tar.gz
-rw-r--r--    1 root     root           566 Jul 24  2023 mattermost-plugin-zoom-v1.6.0-linux-amd64.tar.gz.sig
-rw-r--r--    1 root     root      15339676 Jul 24  2023 mattermost-plugin-jira-v3.2.2-linux-amd64.tar.gz
-rw-r--r--    1 root     root           566 Jul 24  2023 mattermost-plugin-jira-v3.2.2-linux-amd64.tar.gz.sig
-rw-r--r--    1 root     root       9824002 Jul 24  2023 mattermost-plugin-jitsi-v2.0.1-linux-amd64.tar.gz
-rw-r--r--    1 root     root           566 Jul 24  2023 mattermost-plugin-jitsi-v2.0.1-linux-amd64.tar.gz.sig
-rw-r--r--    1 root     root      12526813 Jul 24  2023 mattermost-plugin-nps-v1.3.1-linux-amd64.tar.gz
-rw-r--r--    1 root     root           566 Jul 24  2023 mattermost-plugin-nps-v1.3.1-linux-amd64.tar.gz.sig
-rw-r--r--    1 root     root      11364872 Jul 24  2023 mattermost-plugin-todo-v0.6.1-linux-amd64.tar.gz
-rw-r--r--    1 root     root           566 Jul 24  2023 mattermost-plugin-todo-v0.6.1-linux-amd64.tar.gz.sig
-rw-r--r--    1 root     root      14694197 Jul 24  2023 mattermost-plugin-gitlab-v1.6.0-linux-amd64.tar.gz
-rw-r--r--    1 root     root           566 Jul 24  2023 mattermost-plugin-gitlab-v1.6.0-linux-amd64.tar.gz.sig
-rw-r--r--    1 root     root       9748909 Jul 24  2023 mattermost-plugin-jenkins-v1.1.0-linux-amd64.tar.gz
-rw-r--r--    1 root     root           566 Jul 24  2023 mattermost-plugin-jenkins-v1.1.0-linux-amd64.tar.gz.sig
-rw-r--r--    1 root     root      23653256 Jul 24  2023 mattermost-plugin-playbooks-v1.36.2-linux-amd64.tar.gz
-rw-r--r--    1 root     root           565 Jul 24  2023 mattermost-plugin-playbooks-v1.36.2-linux-amd64.tar.gz.sig
-rw-r--r--    1 root     root      11495404 Jul 24  2023 mattermost-plugin-channel-export-v1.0.0-linux-amd64.tar.gz
-rw-r--r--    1 root     root           566 Jul 24  2023 mattermost-plugin-channel-export-v1.0.0-linux-amd64.tar.gz.sig
-rw-r--r--    1 root     root      10910438 Jul 24  2023 mattermost-plugin-confluence-v1.3.0-linux-amd64.tar.gz
-rw-r--r--    1 root     root           566 Jul 24  2023 mattermost-plugin-confluence-v1.3.0-linux-amd64.tar.gz.sig
-rw-r--r--    1 root     root       8091156 Jul 24  2023 mattermost-plugin-custom-attributes-v1.3.1-linux-amd64.tar.gz
-rw-r--r--    1 root     root           566 Jul 24  2023 mattermost-plugin-custom-attributes-v1.3.1-linux-amd64.tar.gz.sig
-rw-r--r--    1 root     root      15195119 Jul 24  2023 mattermost-plugin-github-v2.1.5-linux-amd64.tar.gz
-rw-r--r--    1 root     root           566 Jul 24  2023 mattermost-plugin-github-v2.1.5-linux-amd64.tar.gz.sig
-rw-r--r--    1 root     root       9039024 Jul 24  2023 mattermost-plugin-antivirus-v0.1.2-linux-amd64.tar.gz
-rw-r--r--    1 root     root           566 Jul 24  2023 mattermost-plugin-antivirus-v0.1.2-linux-amd64.tar.gz.sig
-rw-r--r--    1 root     root       7858736 Jul 24  2023 mattermost-plugin-autolink-v1.2.2-linux-amd64.tar.gz
-rw-r--r--    1 root     root           566 Jul 24  2023 mattermost-plugin-autolink-v1.2.2-linux-amd64.tar.gz.sig
-rw-r--r--    1 root     root      11968541 Jul 24  2023 mattermost-plugin-aws-SNS-v1.2.0-linux-amd64.tar.gz
-rw-r--r--    1 root     root           566 Jul 24  2023 mattermost-plugin-aws-SNS-v1.2.0-linux-amd64.tar.gz.sig
-rw-r--r--    1 root     root      20665013 Jul 24  2023 mattermost-plugin-calls-v0.15.1-linux-amd64.tar.gz
-rw-r--r--    1 root     root           566 Jul 24  2023 mattermost-plugin-calls-v0.15.1-linux-amd64.tar.gz.sig

(yes, I’m running mmost as root)

Relevant part from config.json:

"PluginSettings": {
    "Enable": true,
    "EnableUploads": false,
    "AllowInsecureDownloadURL": false,
    "EnableHealthCheck": true,
    "Directory": "/mattermost/plugins",
    "ClientDirectory": "/mattermost/client/plugins",
    "Plugins": {
        "com.mattermost.plugin-incident-management": {
            "BotUserID": "userid"
        },
        "playbooks": {
            "BotUserID": "userid"
        }
    },
    "PluginStates": {
        "com.mattermost.apps": {
            "Enable": true
        },
        "com.mattermost.calls": {
            "Enable": true
        },
        "com.mattermost.nps": {
            "Enable": true
        },
        "com.mattermost.plugin-incident-management": {
            "Enable": false
        },
        "focalboard": {
            "Enable": true
        },
        "playbooks": {
            "Enable": true
        }
    },
    "EnableMarketplace": true,
    "EnableRemoteMarketplace": true,
    "AutomaticPrepackagedPlugins": true,
    "RequirePluginSignature": false,

Did you start with the Teams (free) version?

Here I’m only referencing the non-enterprise, i.e. Teams version. This one is for my personal usage for friends and family. Workplace uses enterprise, but I’m not managing that one.

And any comment on the collations mismatch above? Should something be done manually there?

7

If you have never used focalboard and don’t plan to, you probably don’t need to bother with the error message.

But you should be aware that we don’t support MariaDB, and by v11, we will stop supporting MySQL as well. So I’d highly suggest to migrate to Postgres.

8

Saw the v10 release notes, already planning to do the migration. Think I first need to fix the error messages related to DB though, which I assume are to do with wrong collations.