Updating Team Edition from 7.5.2 to 7.7.1 results in an error

Troubleshooting
1

After an update the service don’t starts:
Process: 9784 ExecStart=/opt/mattermost/bin/mattermost (code=exited, status=1/FAILURE)
Main PID: 9784 (code=exited, status=1/FAILURE)

…systemd[1]: mattermost.service: Main process exited, code=exited, status=1/FAILURE
…mattermost.service: Failed with result ‘exit-code’.
…Failed to start Mattermost.

The Logfile shows:
{“timestamp”:“2023-02-11 15:44:41.436 +01:00”,“level”:“info”,“msg”:“Server is initializing…”,“caller”:“platform/service.go:165”,“go_version”:“go1.18.1”}
{“timestamp”:“2023-02-11 15:44:41.436 +01:00”,“level”:“info”,“msg”:“Pinging SQL”,“caller”:“sqlstore/store.go:238”,“database”:“master”}
{“timestamp”:“2023-02-11 15:44:41.479 +01:00”,“level”:“info”,“msg”:“Pinging SQL”,“caller”:“sqlstore/store.go:238”,“database”:“master”}
{“timestamp”:“2023-02-11 15:44:41.516 +01:00”,“level”:“info”,“msg”:“Starting websocket hubs”,“caller”:“platform/web_hub.go:95”,“number_of_hubs”:4}
{“timestamp”:“2023-02-11 15:44:41.516 +01:00”,“level”:“info”,“msg”:“Loaded system translations”,“caller”:“i18n/i18n.go:93”,“for locale”:“de”,“from locale”:“/opt/mattermost/i18n/de.json”}
{“timestamp”:“2023-02-11 15:44:41.526 +01:00”,“level”:“info”,“msg”:“Current version is 7.7.1 (7.7.1/Thu Jan 19 00:47:45 UTC 2023/63149c4869898f8c1d3a94ea728370ec4a07575f/none)”,“caller”:“app/server.go:395”,“current_version”:“7.7.1”,“build_number”:“7.7.1”,“build_date”:“Thu Jan 19 00:47:45 UTC 2023”,“build_hash”:“63149c4869898f8c1d3a94ea728370ec4a07575f”,“build_hash_enterprise”:“none”}
{“timestamp”:“2023-02-11 15:44:41.526 +01:00”,“level”:“info”,“msg”:“Team Edition Build”,“caller”:“app/server.go:406”,“enterprise_build”:false}
{“timestamp”:“2023-02-11 15:44:41.526 +01:00”,“level”:“info”,“msg”:“Printing current working”,“caller”:“app/server.go:410”,“directory”:“/opt/mattermost”}
{“timestamp”:“2023-02-11 15:44:41.526 +01:00”,“level”:“info”,“msg”:“Loaded config”,“caller”:“app/server.go:411”,“source”:“file:///opt/mattermost/config/config.json”}
{“timestamp”:“2023-02-11 15:44:41.533 +01:00”,“level”:“info”,“msg”:“Starting workers”,“caller”:“jobs/workers.go:48”}
{“timestamp”:“2023-02-11 15:44:41.534 +01:00”,“level”:“info”,“msg”:“Starting schedulers.”,“caller”:“jobs/schedulers.go:47”}
{“timestamp”:“2023-02-11 15:44:41.546 +01:00”,“level”:“error”,“msg”:“License key from https://mattermost.com required to unlock enterprise features.”,“caller”:“platform/license.go:106”,“error”:“resource: License id: “}
{“timestamp”:“2023-02-11 15:44:41.552 +01:00”,“level”:“fatal”,“msg”:”(app.App).DoPermissionsMigrations failed”,“caller”:“app/migrations.go:555”,“error”:"doPermissionsMigration: The role was not valid., invalid input: entity: Role field: value: &{6jxf58cz6py5drua5amud4f7sr system_admin authentication.Roles.global_admin.name authentication.Roles.global_admin.description 1583839393811 1651073532071 0 [sysconsole_write_compliance create_post_bleve_indexes_job read_data_retention_job sysconsole_read_user_management_teams read_public_channel_groups sysconsole_write_authentication_signup playbook_public_view create_post manage_public_channel_properties test_ldap sysconsole_write_experimental_feature_flags sysconsole_write_experimental_bleve manage_team read_audits recycle_database_connections manage_jobs read_others_bots run_manage_members manage_team_Roles sysconsole_read_site_customization sysconsole_write_site_public_links manage_private_channel_members sysconsole_write_reporting_site_statistics sysconsole_read_billing sysconsole_write_site_customization remove_user_from…
…
This output repeats until I stop the service.

The Schema seems to be on 100 but something goes wrong.

Any ideas?

Platform is ubuntu 20.04 with mysql 8.0.32.

Thanks in advance

Micha

2

Hey fellow mattermoster,

no advices? :disappointed_relieved: Maybe a tipp like how I can restrict the source of this problem?

Hopefully,
Micha

3

What I noticed now: I openend the Systemschema in the system console → user management → permissions. I never changed anything here before (we have the team edition only). I clicked “reset to defaults” and acknowledged the notification. After click on “safe” a red error text appears: “The role was invalid” and i can’t safe my reset try. This is obviously the cause of my update failure.
What can I do now to “reset” the (untouched) roles? The system schema has the 5 roles “all members”, “channel admins”, “playbook admins”, “team admins” and “system admins”.

Should i make a fresh installation of my 7.5.2 version and compare the database table?

After setting the log level to debug and try to save a changed setting in the system schema the log file:

{“timestamp”:“2023-02-16 15:07:08.565 +01:00”,“level”:“debug”,“msg”:“The role was not valid.”,“caller”:“web/context.go:113”,“path”:“/api/v4/roles/6kqwixe757dz7qbohbtfm4pwiy/patch”,“request_id”:“e9uzd8gxzt83pjpyaonce3yrky”,“ip_addr”:“127.0.0.1”,“user_id”:“nfyszaodh7ngdn1y5rgtuimtqa”,“method”:“PUT”,“err_where”:“UpdateRole”,“http_code”:400,“error”:“UpdateRole: Die Rolle war ungültig., invalid input: entity: Role field: value: &{6kqwixe757dz7qbohbtfm4pwiy playbook_admin authentication.Roles.playbook_admin.name authentication.Roles.playbook_admin.description 1643219482530 1651073532200 0 [playbook_private_manage_Roles playbook_private_manage_members playbook_private_manage_properties playbook_public_make_private playbook_public_manage_Roles playbook_public_manage_members playbook_public_manage_properties] true true}”}

4

Finally. The problem was, that a sysadmin and a sysadmin are not the same.
The first sysadmin account on my mattermost installation I rarely used anymore after the installation and authorized my personal user as a sysadmin to do the admin work in sysconsole and so on.

Today I tried to login with the original admin account and acknowledged a bunch of “welcome to the new feature” windows - one of them for some kind of playbook news. After that I opened the system console and tried to change the system scheme again. This time without errors!

Long story short - also the update to (meanwhile) 7.8.0 was succesful and without errors.

5

Hi @Micha , welcome to the Mattermost forums and sorry for the late reply!

Glad to hear you got your problem fixed, although I’m wondering what the underlying issue was - do you have access to the database of your installation? If so, could you please run the following query:

select username,roles,props FROM users where username IN ('user1', 'user2');

where user1 and user2 are the two usernames in question (the initial sysadmin and your account that you made sysadmin).

Also could you post the output of this query:

select permissions from roles where name='system_admin';

Could it be that in the past, you once migrated to the enterprise build and then switched back to teams edition? Not sure if this is really a thing here, but getting “the role was not valid” in the server logs indicates that there’s something in the database that might look strange.

I assume you do not have a copy of the database before you tried to run the upgrade, do you?

6

Hey, nice to read from you. All good.

The noticeable in the 2 lines output of the first statement is (I leave off the customstatus stuff in the props field) that the initial sysadmin has in the roles field “system_admin system_user” and my account “system_user system_admin”.

The output of your 2nd statement is

 view_members sysconsole_write_environment_push_notification_server playbook_private_manage_members create_ldap_sync_job sysconsole_write_environment_rate_limiting playbook_public_manage_properties sysconsole_write_environment_image_proxy sysconsole_write_site_customization sysconsole_write_environment_web_server create_elasticsearch_post_indexing_job sysconsole_write_compliance_custom_terms_of_service sysconsole_write_site_file_sharing_and_downloads sysconsole_read_about_edition_and_license read_public_channel reload_config read_user_access_token sysconsole_write_authentication_password sysconsole_read_environment_smtp read_channel convert_public_channel_to_private sysconsole_read_site_notices create_user_access_token sysconsole_write_authentication_mfa sysconsole_read_environment_database sysconsole_read_environment_elasticsearch create_private_channel sysconsole_read_environment_high_availability sysconsole_write_reporting_team_statistics remove_ldap_private_cert read_elasticsearch_post_indexing_job manage_secure_connections edit_brand create_emojis manage_outgoing_webhooks get_saml_cert_status sysconsole_write_authentication_saml manage_team read_audits sysconsole_write_compliance_data_retention_policy manage_others_slash_commands remove_others_reactions sysconsole_read_site_notifications sysconsole_write_environment_database add_reaction delete_emojis sysconsole_write_environment_developer invite_user sysconsole_read_billing sysconsole_read_products_boards sysconsole_read_compliance_compliance_monitoring test_email sysconsole_read_integrations_integration_management convert_private_channel_to_public sysconsole_write_integrations_integration_management purge_elasticsearch_indexes manage_others_bots sysconsole_read_user_management_teams sysconsole_read_compliance_data_retention_policy playbook_private_view read_other_users_teams sysconsole_write_user_management_teams sysconsole_read_authentication_email sysconsole_write_site_announcement_banner sysconsole_read_authentication_password sysconsole_read_site_customization sysconsole_read_environment_performance_monitoring get_saml_metadata_from_idp sysconsole_write_experimental_feature_flags purge_bleve_indexes edit_others_posts manage_roles sysconsole_read_plugins manage_public_channel_members manage_public_channel_properties create_post_public sysconsole_read_environment_web_server playbook_public_create read_deleted_posts sysconsole_read_environment_image_proxy delete_others_posts playbook_public_manage_roles sysconsole_write_authentication_signup read_license_information join_public_channels playbook_private_make_public manage_license_information sysconsole_write_site_users_and_teams sysconsole_write_environment_smtp sysconsole_read_environment_developer invalidate_caches sysconsole_read_integrations_bot_accounts sysconsole_write_integrations_gif read_others_bots sysconsole_write_environment_session_lengths list_team_channels join_private_teams manage_shared_channels sysconsole_read_site_localization sysconsole_write_experimental_features manage_others_incoming_webhooks add_saml_idp_cert manage_oauth test_s3 sysconsole_write_compliance_compliance_export playbook_private_manage_roles sysconsole_read_compliance_custom_terms_of_service sysconsole_write_compliance_compliance_monitoring sysconsole_read_experimental_bleve get_logs import_team sysconsole_write_user_management_groups sysconsole_read_reporting_server_logs view_team sysconsole_read_site_public_links sysconsole_read_reporting_team_statistics sysconsole_read_integrations_gif list_private_teams use_slash_commands run_create sysconsole_read_experimental_features test_elasticsearch manage_private_channel_properties sysconsole_write_integrations_cors sysconsole_write_site_posts sysconsole_write_plugins sysconsole_write_user_management_users sysconsole_read_environment_logging playbook_private_create sysconsole_write_reporting_server_logs read_compliance_export_job sysconsole_write_integrations_bot_accounts sysconsole_read_environment_rate_limiting sysconsole_read_authentication_saml get_analytics create_elasticsearch_post_aggregation_job sysconsole_read_site_announcement_banner sysconsole_read_site_emoji sysconsole_read_environment_push_notification_server add_saml_private_cert sysconsole_read_authentication_mfa create_compliance_export_job manage_system sysconsole_write_reporting_site_statistics playbook_private_manage_properties manage_system_wide_oauth run_manage_properties read_ldap_sync_job add_ldap_private_cert read_public_channel_groups edit_post manage_custom_group_members sysconsole_write_environment_high_availability join_public_teams delete_public_channel create_data_retention_job sysconsole_read_environment_file_storage sysconsole_write_authentication_openid restore_custom_group sysconsole_write_environment_performance_monitoring read_bots sysconsole_write_experimental_bleve read_private_channel_groups sysconsole_read_environment_session_lengths invalidate_email_invite sysconsole_read_experimental_feature_flags read_jobs playbook_public_make_private test_site_url delete_custom_group delete_private_channel add_user_to_team delete_post sysconsole_write_billing manage_bots sysconsole_write_environment_elasticsearch remove_saml_private_cert playbook_public_manage_members demote_to_guest promote_guest manage_slash_commands sysconsole_read_authentication_ldap list_public_teams create_post recycle_database_connections use_channel_mentions sysconsole_read_authentication_openid create_team remove_ldap_public_cert sysconsole_read_reporting_site_statistics remove_user_from_team sysconsole_write_user_management_system_roles list_users_without_team remove_reaction assign_system_admin_role sysconsole_write_environment_file_storage sysconsole_write_site_localization sysconsole_write_site_emoji sysconsole_read_site_posts add_saml_public_cert sysconsole_write_environment_logging sysconsole_write_about_edition_and_license sysconsole_read_site_file_sharing_and_downloads read_elasticsearch_post_aggregation_job sysconsole_write_authentication_guest_access sysconsole_write_site_notices sysconsole_write_authentication_ldap sysconsole_write_products_boards download_compliance_export_result sysconsole_read_authentication_signup edit_custom_group create_direct_channel create_group_channel manage_channel_roles sysconsole_write_site_public_links read_data_retention_job sysconsole_read_authentication_guest_access test_ldap sysconsole_read_user_management_permissions sysconsole_read_integrations_cors manage_private_channel_members playbook_public_view add_ldap_public_cert create_post_ephemeral sysconsole_read_compliance_compliance_export assign_bot sysconsole_write_user_management_channels manage_others_outgoing_webhooks remove_saml_idp_cert revoke_user_access_token sysconsole_write_user_management_permissions sysconsole_write_site_notifications invite_guest create_bot create_post_bleve_indexes_job create_custom_group create_public_channel manage_jobs manage_team_roles sysconsole_write_authentication_email manage_incoming_webhooks upload_file run_manage_members run_view sysconsole_read_site_users_and_teams sysconsole_read_user_management_channels sysconsole_read_user_management_groups delete_others_emojis use_group_mentions sysconsole_read_user_management_users remove_saml_public_cert sysconsole_read_user_management_system_roles get_public_link edit_other_users

Oh, your assumption of some “history” of the database and it’s tables is right. I will not rule out that I at one update run in the past 2 years inadvertently downloaded the enterprise package and tried an update. But in addition we started with mattermost on a windows 2012 server with mysql running on it. Because of failed update tries some time ago I went through the adventure to migrate the database to an installation on a supported ubuntu vm. I had to export/import the database, ran through problems like differences in all 82 table names because lowercase table names in windows mysql and upper case initial letter in linux, followed by duplicate keys in some tables and so on.
But at the end all was ok and a few update runs where successful. Till 7.5.2 → 7.7.1.