Mattermost board as product does not work

Troubleshooting
1

Summary
On self hosted mattermost installation, our “Board” is installed as product (feature flag BoardsProduct = true).
But when accessing to board we’ve error “Something went wrong”

Steps to reproduce/Expected behavior
Unfortunately I didn’t do the installation so I don’t have the detail.
But mostly I’d like to understand what is the diff between Product and no Product mode?
And can I switch to no product mode?

Observed behavior
Every calls to

https://MATTERMOST/plugins/boards/ returns 404

like https://MATTERMOST/plugins/boards/api/v2/users/me

2

Hi @kakawait ,

what’s your server version? Boards in product mode is only supported on the newer server versions.
Also I would be interested in the most recent lines in your mattermost.log file immediately after server start or after you enabled the boards plugin in your system console.

3

Thank you for your help.

I’m using docker-compose version based on

MATTERMOST_IMAGE_TAG=release-7.7 (official image)

The log when I try to enable the Mattermost board plugin

mattermost-mattermost-1  | {"timestamp":"2023-01-23 16:42:23.527 Z","level":"error","msg":"Plugin focalboard cannot be enabled in product mode.","caller":"web/context.go:117","path":"/api/v4/config","request_id":"acx348kge78zfct86terymtmer","ip_addr":"172.19.0.1","user_id":"4t75ktcf5fyomgra4xjn9ej9ey","method":"PUT","err_where":"EnablePlugin","http_code":500,"error":"EnablePlugin: Plugin focalboard cannot be enabled in product mode."}
4

OK, then boards is already running in product mode and should also be active already, but there might be some migration issues which prevent it from working properly.

Do you still have the logfiles of the server startup available or can you restart the server and generate new logs? I’m especially interested in anything that looks like it’s related to the boards feature.

5

I can’t restart ATM people using it but here the full log.
I saw database init so I think it should contains startup.

Is recent installation (dec 2022 / jan 2023).

Tell me is enough?

The only specific settings is we’re using GitLab (self hosted auth)

PS: I see some SQL errors due to UK constraint issue, I don’t know if relevant

6

Thanks, I can see a duplicate key constraint error in your database:

mattermost-postgres-1  | 2023-01-18 10:59:31.986 UTC [15088] ERROR:  duplicate key value violates unique constraint "channels_name_teamid_key"
mattermost-postgres-1  | 2023-01-18 10:59:31.986 UTC [15088] DETAIL:  Key (name, teamid)=(ux-team, rp31g6k96b8oiktc5ze46kmxya) already exists.
mattermost-postgres-1  | 2023-01-18 10:59:31.986 UTC [15088] STATEMENT:  INSERT INTO Channels
mattermost-postgres-1  | 			(Id, CreateAt, UpdateAt, DeleteAt, TeamId, Type, DisplayName, Name, Header, Purpose, LastPostAt, TotalMsgCount, ExtraUpdateAt, CreatorId, SchemeId, GroupConstrained, Shared, TotalMsgCountRoot, LastRootPostAt)
mattermost-postgres-1  | 			VALUES
mattermost-postgres-1  | 			($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19)

Please connect to your PostgreSQL DB and run the following query there in your mattermost database:

select id,createat,deleteat,teamid,type,displayname,name from channels where name IN ('town-square', '164ceb086f907e42b015de1395bccc2e5790f04c');

Let’s see if we can fix the database when we see what is in it.

While you are connected, please do also run the following queries so we can get an idea about the current state of your focalboard deployment:

\dt focalboard*
select * from focalboard_schema_migrations;
7

@agnivade Could this be related to [MM-49790] - Mattermost?

8

I can see the following line in the logs which should not show up for teams builds AFAIK:

mattermost-mattermost-1  | {"timestamp":"2023-01-13 12:52:47.764 Z","level":"error","msg":"License key from https://mattermost.com required to unlock enterprise features.","caller":"platform/license.go:106","error":"resource: License id: "}
9 
mattermost=# select id,createat,deleteat,teamid,type,displayname,name from channels where name IN ('town-square', '164ceb086f907e42b015de1395bccc2e5790f04c');
             id             |   createat    | deleteat |           teamid           | type |   displayname   |                   name                   
----------------------------+---------------+----------+----------------------------+------+-----------------+------------------------------------------
 nb18bmzq6tf65fg9tkcryiduta | 1673016052238 |        0 | 64i4f7eh3pdxjkfehyrxh33b3w | O    | Town Square     | town-square
 qm9i3mta8fgydx6ds7uinw4hze | 1673023018311 |        0 | opx8nxywsj88mm96oy7mqgnwac | O    | Général         | town-square
 7hc9p4u7qbgm5ptwjew9q7w97r | 1673023026380 |        0 | kwq6eusyx38hzgf1ijyw7z5hio | O    | Général         | town-square
 xeobqfp9ffy5i8rmhym7r9qhoh | 1673016039019 |        0 | rp31g6k96b8oiktc5ze46kmxya | O    | Général 🇫🇷      | town-square
 i694dro7g7g59k4qptthz543gw | 1673022944209 |        0 | ib4tg877jfdyijzsfwk8wqi3so | O    | Town Square     | town-square
 nmfx68mp438g9xd753rhiop5nw | 1673250151653 |        0 | 3xasrdsatpnfukd4pccdooeawy | O    | Town Square ❤️   | town-square
 jinaoatoytbfm8bddwdx4wjr6y | 1674140500548 |        0 |                            | G    | USER1, USER2, USER3 | 164ceb086f907e42b015de1395bccc2e5790f04c
(7 rows)

PS: I’ve only edited USER1, USER2, USER3 which a user naming

mattermost=# \dt focalboard*
                       List of relations
 Schema |               Name               | Type  |   Owner    
--------+----------------------------------+-------+------------
 public | focalboard_blocks                | table | mattermost
 public | focalboard_blocks_history        | table | mattermost
 public | focalboard_board_members         | table | mattermost
 public | focalboard_board_members_history | table | mattermost
 public | focalboard_boards                | table | mattermost
 public | focalboard_boards_history        | table | mattermost
 public | focalboard_categories            | table | mattermost
 public | focalboard_category_boards       | table | mattermost
 public | focalboard_file_info             | table | mattermost
 public | focalboard_notification_hints    | table | mattermost
 public | focalboard_preferences           | table | mattermost
 public | focalboard_schema_migrations     | table | mattermost
 public | focalboard_sessions              | table | mattermost
 public | focalboard_sharing               | table | mattermost
 public | focalboard_subscriptions         | table | mattermost
 public | focalboard_system_settings       | table | mattermost
 public | focalboard_teams                 | table | mattermost
 public | focalboard_users                 | table | mattermost
(18 rows)

mattermost=# select * from focalboard_schema_migrations;
 version |                name                 
---------+-------------------------------------
       1 | init
       2 | system_settings_table
       3 | blocks_rootid
       4 | auth_table
       5 | blocks_modifiedby
       6 | sharing_table
       7 | workspaces_table
       8 | teams
       9 | blocks_history
      10 | blocks_created_by
      11 | match_collation
      12 | match_column_collation
      13 | millisecond_timestamps
      14 | add_not_null_constraint
      15 | blocks_history_no_nulls
      16 | subscriptions_table
      17 | add_file_info
      18 | add_teams_and_boards
      19 | populate_categories
      20 | populate_category_blocks
      21 | create_boards_members_history
      22 | create_default_board_role
      23 | persist_category_collapsed_state
      24 | mark_existsing_categories_collapsed
      25 | indexes_update
      26 | create_preferences_table
      27 | migrate_user_props_to_preferences
      28 | remove_template_channel_link
      29 | add_category_type_field
(29 rows)

For build version, it may be possible we start with enterprise (because was the default value on .env.sample) then switched to team

Because I’m using

mattermost-mattermost-1   mattermost/mattermost-team-edition:release-7.7   "/entrypoint.sh bash…"   mattermost          10 days ago         Up 31 minutes (healthy)   8067/tcp, 0.0.0.0:8065->8065/tcp, :::8065->8065/tcp, 8074-8075/tcp, 0.0.0.0:8443->8443/udp, :::8443->8443/udp
10

OK, can you - for testing purposes and to validate @amy.blais idea switch back to the enterprise version for the time being?
In order to get boards fixed, you will have to restart the server anyways, either by setting the environment variable MM_FEATUREFLAGS_BoardsProduct=false on the teams version or by switching back to the enterprise version. I’d suggest to switch back to the enterprise version for the time being and wait until a fix is out there for the teams edition then so you can then safely switch back without any impact to the boards plugin.

With regards to your team names, I think this is related to the emojis in the channel names which seem to be not properly taken into consideration for the uniqueness, but I’m again unsure why.

To verify that the collations are correct, can you please provide the output of:

\l mattermost
\d+ channels
11

Thank you for all support! Really great spirit of helping and opensource at GitLab and Mattermost.

I’ll not able to do such operation until tomorrow night (GMT+1)

I’ll keep you in touch

12

@kakawait - Was there a reason you set the feature flag BoardsProduct to true? I don’t believe this was documented anywhere in the release notes. Please set that to false and you should be back to running.

13

I didn’t. I’ll check with other administrator.

14

I’ve tested enterprise edition but does do anything special.
Thus I’ve used MM_FEATUREFLAGS_BoardsProduct=false and then re-install plugin (Mattermost boards) from marketplace. Now it works !

I don’t know where or who has set the flag to true I’ll try to invest. But I’m curious that come from the other administrator.

@agriesser I don’t see any DUP error on startup anymore (I did nothing for that) but I could consider is ok? Or regarding migration steps I’ve posted, should I’ve missed some?

But in case, here the output psql

mattermost=# \l mattermost
                                List of databases
    Name    |   Owner    | Encoding |  Collate   |   Ctype    | Access privileges 
------------+------------+----------+------------+------------+-------------------
 mattermost | mattermost | UTF8     | en_US.utf8 | en_US.utf8 | 
(1 row)

mattermost=# \d+ channels
                                                 Table "public.channels"
      Column       |          Type           | Collation | Nullable |   Default   | Storage  | Stats target | Description 
-------------------+-------------------------+-----------+----------+-------------+----------+--------------+-------------
 id                | character varying(26)   |           | not null |             | extended |              | 
 createat          | bigint                  |           |          |             | plain    |              | 
 updateat          | bigint                  |           |          |             | plain    |              | 
 deleteat          | bigint                  |           |          |             | plain    |              | 
 teamid            | character varying(26)   |           |          |             | extended |              | 
 type              | channel_type            |           |          |             | plain    |              | 
 displayname       | character varying(64)   |           |          |             | extended |              | 
 name              | character varying(64)   |           |          |             | extended |              | 
 header            | character varying(1024) |           |          |             | extended |              | 
 purpose           | character varying(250)  |           |          |             | extended |              | 
 lastpostat        | bigint                  |           |          |             | plain    |              | 
 totalmsgcount     | bigint                  |           |          |             | plain    |              | 
 extraupdateat     | bigint                  |           |          |             | plain    |              | 
 creatorid         | character varying(26)   |           |          |             | extended |              | 
 schemeid          | character varying(26)   |           |          |             | extended |              | 
 groupconstrained  | boolean                 |           |          |             | plain    |              | 
 shared            | boolean                 |           |          |             | plain    |              | 
 totalmsgcountroot | bigint                  |           |          |             | plain    |              | 
 lastrootpostat    | bigint                  |           |          | '0'::bigint | plain    |              | 
Indexes: 
    "channels_pkey" PRIMARY KEY, btree (id)
    "channels_name_teamid_key" UNIQUE CONSTRAINT, btree (name, teamid)
    "idx_channel_search_txt" gin (to_tsvector('english'::regconfig, (((name::text || ' '::text) || displayname::text) || ' '::text) || purpose::text))
    "idx_channels_create_at" btree (createat)
    "idx_channels_delete_at" btree (deleteat)
    "idx_channels_displayname_lower" btree (lower(displayname::text))
    "idx_channels_name_lower" btree (lower(name::text))
    "idx_channels_scheme_id" btree (schemeid)
    "idx_channels_team_id_display_name" btree (teamid, displayname)
    "idx_channels_team_id_type" btree (teamid, type)
    "idx_channels_update_at" btree (updateat)
Access method: heap

The only error I still have is

mattermost-mattermost-1  | {"timestamp":"2023-01-23 22:00:40.455 Z","level":"error","msg":"License key from https://mattermost.com required to unlock enterprise features.","caller":"platform/license.go:106","error":"resource: License id: "}
mattermost-mattermost-1  | {"timestamp":"2023-01-23 22:00:40.899 Z","level":"error","msg":"Failed to sync plugin from file store","caller":"app/plugin.go:373","bundle":"plugins/mattermost-autolink.tar.gz","error":"installExtractedPlugin: Unable to restart plugin on upgrade., unable to start plugin: mattermost-autolink: fork/exec plugins/mattermost-autolink/server/dist/plugin-linux-amd64: text file busy"}

I think the first one can be ignored. I’ll check for autolink, but plugin work correctly

15

Hi again,

the duplicate key errors did not get posted during startup, they maybe get posted when a specific action is being carried out, but I’m not sure which one. If they never show up again, we’re good to go, but still it looks strange to me. The collations in your database look good, though.

With regards to the remaining error messages:

This one just indicates that there’s an enterprise build running without a license. It can be ignored, if you want to stay on the enterprise version. As you pointed out, there’s no difference feature-wise between the unlicensed enterprise edition build and the teams edition build.

This one means that there is a version of the plugin in the local filestore that cannot be replaced during startup, because the process is already running (each plugin starts as its own daemon).
You can try to manually update the plugin using the Marketplace if it’s outdated and if this throws the same error or still doesn’t work, you can try to disable it first in the system console’s plugin management and try the upgrade again, then (after confirming that the binary is not running anymore using ps ax on the linux console).

16

In a self-hosted installation of v7.7 I was seeing this error message when attempting to save settings - they wouldn’t save.

Plugin focalboard cannot be enabled in product mode.`

I eventually tracked this down to me turning on a setting that I shouldn’t have. There is a setting to turn on boards, but since v7.7 includes boards by default this setting is not needed to be turned on.

System Console
PLUGINS → Mattermost Boards → Enable Plugin

I had turned this setting to true, but it needs to be false.

17

Hi @PromInc, welcome to the Mattermost forums and thanks for the additional information!

You’re right, this is a non-default setting that causes troubles on upgrades recently and it is being discussed that the server application will be more tolerant about that in the future. For now, your solution works (this setting can also be toggled directly in config.json, when you search for focalboard there).