Couldn't upload profile image & Playbooks plugin failing

Summary
Couldn’t upload profile image errors showing which seems to also prevent playbook plugin from working.

Steps to reproduce
Using Docker deployment & latest tag.
Initial deployment used the docker images for pg, mattermost, and nginx (with nginx.yml). Image upload worked as expected, but test URL continued to fail and desktop client would not connect. Playbooks option in menu was present.
I removed nginx and turned up MM with the “…without.nginx.yml” compose file, and deployed nginx on the host.
TLS in connection security is enabled. Test URL and client app can now connect and pass. File storage is set to AWS and tests pass.
Now Playbook option is missing, and errors show when trying to change profile pic.

Expected behavior
Profile pics upload successfully. Playbooks are enabled and menu option present.

Observed behavior
Errors when trying to upload profile pics and as a result, the playbook plugin is complaining about profile pic change fail as well.
Error message for profle pic:

{
  "caller": "web/context.go:115",
  "err_where": "SetProfileImage",
  "error": "SetProfileImage: Couldn't upload profile image., WriteFile: Unable to write the file., unable write the data in the file users/mp9nxxb8xpbsdkouyrsepzeqko/profile.png: User: arn:aws:iam::680983143169:user/iq-mmost-s3 is not authorized to perform: kms:GenerateDataKey on resource: arn:aws:kms:us-west-2:680983143169:key/5cb4106c-4be0-4488-ab05-f3df1b5c200e because no identity-based policy allows the kms:GenerateDataKey action",
  "http_code": 500,
  "ip_addr": "172.31.0.1",
  "level": "error",
  "method": "POST",
  "msg": "Couldn't upload profile image.",
  "path": "/api/v4/users/mp9nxxb8xpbsdkouyrsepzeqko/image",
  "request_id": "um4gz4qibby5iq1xzzk6ewe9yc",
  "timestamp": "2023-12-13 00:05:32.158 Z",
  "user_id": "mp9nxxb8xpbsdkouyrsepzeqko"
}

Errors on Playbook:
1

{
  "bundle_path": "/mattermost/prepackaged_plugins/mattermost-plugin-playbooks-v1.39.1-linux-amd64.tar.gz",
  "caller": "app/plugin.go:966",
  "error": "Failed to install extracted prepackaged plugin /mattermost/prepackaged_plugins/mattermost-plugin-playbooks-v1.39.1-linux-amd64.tar.gz: installExtractedPlugin: Unable to restart plugin on upgrade., failed to ensure bot: failed to set profile image: SetProfileImage: Couldn't upload profile image.",
  "level": "error",
  "msg": "Failed to install prepackaged plugin",
  "timestamp": "2023-12-13 00:00:19.756 Z"
}

2

{
  "bundle_path": "plugins/playbooks",
  "caller": "app/plugin.go:171",
  "error": "failed to ensure bot: failed to set profile image: SetProfileImage: Couldn't upload profile image.",
  "level": "error",
  "msg": "Unable to activate plugin",
  "plugin_id": "playbooks",
  "timestamp": "2023-12-13 00:04:50.995 Z"
}

Docker Compose Files:

# https://docs.docker.com/compose/environment-variables/

version: "2.4"

services:
  postgres:
    image: postgres:${POSTGRES_IMAGE_TAG}
    restart: ${RESTART_POLICY}
    security_opt:
      - no-new-privileges:true
    pids_limit: 100
    read_only: true
    tmpfs:
      - /tmp
      - /var/run/postgresql
    volumes:
      - ${POSTGRES_DATA_PATH}:/var/lib/postgresql/data
    environment:
      # timezone inside container
      - TZ

      # necessary Postgres options/variables
      - POSTGRES_USER
      - POSTGRES_PASSWORD
      - POSTGRES_DB

  mattermost:
    depends_on:
      - postgres
    image: mattermost/${MATTERMOST_IMAGE}:${MATTERMOST_IMAGE_TAG}
    restart: ${RESTART_POLICY}
    security_opt:
      - no-new-privileges:true
    pids_limit: 200
    read_only: ${MATTERMOST_CONTAINER_READONLY}
    tmpfs:
      - /tmp
    ports:
      - 8065:8065
    volumes:
      - ${MATTERMOST_CONFIG_PATH}:/mattermost/config:rw
      - ${MATTERMOST_DATA_PATH}:/mattermost/data:rw
      - ${MATTERMOST_LOGS_PATH}:/mattermost/logs:rw
      - ${MATTERMOST_PLUGINS_PATH}:/mattermost/plugins:rw
      - ${MATTERMOST_CLIENT_PLUGINS_PATH}:/mattermost/client/plugins:rw
      - ${MATTERMOST_BLEVE_INDEXES_PATH}:/mattermost/bleve-indexes:rw
      - ${CERT_PATH}:/cert.pem:ro
      - ${KEY_PATH}:/key.pem:ro

      # When you want to use SSO with GitLab, you have to add the cert pki chain of GitLab inside Alpine
      # to avoid Token request failed: certificate signed by unknown authority
      # (link: https://github.com/mattermost/mattermost-server/issues/13059 and https://github.com/mattermost/docker/issues/34)
      # - ${GITLAB_PKI_CHAIN_PATH}:/etc/ssl/certs/pki_chain.pem:ro
    environment:
      # timezone inside container
      - TZ

      # necessary Mattermost options/variables (see env.example)
      - MM_SQLSETTINGS_DRIVERNAME
      - MM_SQLSETTINGS_DATASOURCE

      # necessary for bleve
      - MM_BLEVESETTINGS_INDEXDIR

      # additional settings
      - MM_SERVICESETTINGS_SITEURL

# If you use rolling image tags and feel lucky watchtower can automatically pull new images and
# instantiate containers from it. https://containrrr.dev/watchtower/
# Please keep in mind watchtower will have access on the docker socket. This can be a security risk.
#
#  watchtower:
#    container_name: watchtower
#    image: containrrr/watchtower:latest
#    restart: unless-stopped
#    volumes:
#      - /var/run/docker.sock:/var/run/docker.sock

Without nginx compose file:

version: "2.4"

services:
  mattermost:
    ports:
      - ${APP_PORT}
     # - ${APP_PORT}:8065
      - ${CALLS_PORT}:${CALLS_PORT}/udp
      - ${CALLS_PORT}:${CALLS_PORT}/tcp

I cannot recall if the profile pic worked originally when set to local files vs if I switched to AWS before or after the initial time it worked. I changed file storage back to local and the profile upload failed again. I’d love any ideas on what to look at to correct.
Thanks

I reviewed the error log for the image upload and noted the permissions on AWS were not set to allow using the KMS encryption. I amended the policy to the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket>"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket>/*"
            ]
        },
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Action": [
                "kms:Encrypt",
                "kms:ReEncryptTo",
                "kms:Decrypt",
                "kms:GenerateDataKey",
                "kms:DescribeKey"
            ],
            "Resource": [
                "arn:aws:kms:region:account-id:key/key-id"
            ]
        }
    ]
}

Now uploads (images and filles) work as expected. I restarted the server and the Playbook plugin is still failing - showing the same two error messages, one referring to:
“error”: “failed to ensure bot: failed to set profile image: SetProfileImage: Couldn’t upload profile image.”,

Any ideas on how to correct or find the issue?

I restarted the containers and now I am back to the file upload errors, with a new message:
“Requests specifying Server Side Encryption with AWS KMS managed keys must be made over a secure connection.”
No changes were made, only the service (Docker container) restarted. TLS is setup for all connections, and no errors showing in the logs on that.

I went ahead and deleted the KMS Encryption on S3 and will use Amazon S3 managed keys (SSE-S3). This seems to have resolved the upload and playbooks issue.
Hopefully it will help others in case they set KMS on the bucket.