Mattermost v7.9.1 team-edition helm deployment issues

Troubleshooting
1

Env:
AWS EKS Cluster running v1.23 Kubernetes
Using standard StorageClass gp2 provided by the EKS deployment

I am using the full values file for deployment:

helm install mattermost-server mattermost/mattermost-team-edition -f mattermost-values.yaml -n mattermost --set storageclass=gp2

During deployment, issues with permissions to create directories for plugins.

Error Example:
mattermost-team-edition {“timestamp”:“2023-03-19 04:04:06.131 Z”,“level”:“error”,“msg”:“Failed to install prepackaged plugin”,“caller”:“app/plugin.go:964”,“path”:“/mattermost/prepackaged_plugins/mattermost-plugin-calls-v0.14.0-linux-amd64.tar.gz”,“error”:“Failed to install extracted prepackaged plugin /mattermost/prepackaged_plugins/mattermost-plugin-calls-v0.14.0-linux-amd64.tar.gz: installExtractedPlugin: Unable to generate plugin webapp bundle., unable to copy webapp bundle directory: com.mattermost.calls: mkdir client/plugins/com.mattermost.calls: permission denied”}

mkdir client/plugins/com.mattermost.nps: permission denied"
mkdir client/plugins/playbooks: permission denied"
mkdir client/plugins/com.mattermost.calls: permission denied"
mkdir client/plugins/focalboard: permission denied"

After running exec into the pod, I am also unable to create the directories unless I change the permissions on the plugins directory to 777

There is also a nagging license error

I am prevented from installing gitlab plugin from the Console which is required for our testing.

2

Did you change any policies and/or user & group IDs?

Some policies & user/group IDs are hard-coded. The way to fix it for you individually is to fork the Helm chart, build your own Docker image with the IDs you need & then it should work.

Otherwise, there may be some restricitions on your vendor-specific storage method.

3

Thanks for taking a look!
This is a fresh deployment with no changes to policies or user/group ID’s

4

I changed the deployment to this, so am using all but a few default values
helm install mattermost-server mattermost/mattermost-team-edition -n mattermost --set storageclass=gp2 --set service.type=LoadBalancer --set service.externalPort=80 --set mysql.mysqlUser=mattermost --set mysql.mysqlPassword=abc123

Issues still persists

5

Is it possible this is the problem ?
mattermost-server-mattermost-team-edition-548ddc6f6b-jfrq6 mattermost-team-edition {“timestamp”:“2023-03-20 18:32:21.875 Z”,“level”:“debug”,“msg”:“No license provided; Remote Cluster services disabled”,“caller”:“app/server.go:576”}

6

in the client directory, plugins is the only thing owned by root
drwxr-xr-x 3 root root 4096 Mar 20 19:19 plugins

7

As a work around I have tried getting root into the pod using both kubectl ssh and runc without any luck to change the owner as the the pod is coming up.

Anyone know of another method? I can’t ssh to nodes so thats not an option

8

where can I get a full list of options for MM environment variables ?

9

This list should be quite comprehensive and you’ll see that there’s a scheme behind the environment variable names:

https://docs.mattermost.com/configure/configuration-settings.html

If you want to override the configuration setting TeamSettings.EnableCustomBrand in your config.json with an environment variable, its name is MM_TEAMSETTINGS_ENABLECUSTOMBRAND.

10

That is a known problem with kubectl.

Try chown 1001:1001 -R plugins. Replace 1001 with whatever user the other folders use.

11

By default, anything configured during the deployment for plugins is created with root as the owner.
short of a root login, there is no way to fix this issue. I will assume from this that the teams deployment has disabled plugins as part of version restrictions.

Thanks everyone who has tried to help

12

My last post on the subject. Enterprise when installed without a license still has the correct permissions and deploys the default pulugins.

mattermost@mattermost-ent-mattermost-enterprise-edition-678857cdcf-jhf7n:~$ ls -l client/ | grep plugin
drwxrwxrwx 8 root root 4096 Mar 21 23:13 plugins
mattermost@mattermost-ent-mattermost-enterprise-edition-678857cdcf-jhf7n:~$ cd client/plugins/
mattermost@mattermost-ent-mattermost-enterprise-edition-678857cdcf-jhf7n:~/client/plugins$ ls
com.mattermost.calls com.mattermost.nps com.mattermost.plugin-channel-export focalboard jitsi playbooks
mattermost@mattermost-ent-mattermost-enterprise-edition-678857cdcf-jhf7n:~/client/plugins

So it appears to be a code issue and not much I can do about that.

13

So you’re saying that the enterprise build has no problem with the plugins, but the teamedition does? I’m not sure what you meant with “still has the correct permissions”.

14

I am suggesting its not a license issue but just some code. team-edition does complain about lack of a license.

15

Back door fix:
Make sure you have ssh access to workers in the cluster. I had not addedd rsa key to my EKS deployment and had to add that to get things working.

As the deployment spins up do a kubectl gep pod -o wide and get the node the pod is running on.
log into the the node.
run sudo docker ps | grep edition to get the docker id

run docker exec -u 0 instanceID chmod 777 /mattermost/client/plugins

Add S3 bucket in file storage, restart the pod, this allows the default plugins (including apps) to deploy

kubectl rollout restart deployment/mattermost-server-mattermost-team-edition -n mattermost

16

I don’t know if it will help because my English is not good.

Env:
on-prem , k8s v1.24.11
mattermost team edition v7.9.1, helm chart installed
storage use minio bucket.

i installed based on this guide.
https://docs.mattermost.com/install/installing-team-edition-helm-chart.html

My problem was also a permission problem when installing the plugin.

added env to values.yaml

config:
MM_FILESETTINGS_AMAZONS3SSL: “false”
image
image728×348 10.8 KB

also, can test at mattermost
*System Console - File Storage - Enable Secure Amazon S3 Connections: false

17

Hi @new2001yy and welcome to the Mattermost forums!

Are you using an insecure deployment of minio maybe for your S3 backend? That would explain why you have to disable the SSL security for the S3 backends.

18

I found a workaround. Add this to you helm values:

extraInitContainers:
  - command:
      - sh
      - '-c'
      - chown -R 2000:2000 /client/plugins
    image: busybox
    name: changeowner
    volumeMounts:
      - mountPath: /client/plugins
        name: mattermost-plugins

This runs a init container which changes the owner of the plugins folder.

19

Interesting workaround - thanks for sharing!

20

Thanks! Will take a look at that soon