Mattermost Service on Red Hat 9.5.0 Failing to Start: Operation not permitted

Troubleshooting
1

Good-day Folks,

I trust all is well with you as you read this post. I am very new to Mattermost and I’m trying to get it to work in an air-gapped test lab at work (as proof of concept for wider adoption within my organization). To that effect, I attempted to deploy the latest community edition and the service is failing to start.

My Environment

  • Database: PostgreSQL v17 running on Windows Server 2016
  • Mattermost: v10.3.1 running on Red Hat Linux v9.5.0

Summary
The Mattermost service is failing to start; outputting the following error message:

mattermost.service: Failed to execute /opt/mattermost/bin/mattermost: Operation not permitted
mattermost.service: Failed at step EXEC spawning /opt/mattermost/bin/mattermost: Operation not permitted

Steps to reproduce

  1. Build a VM running Red Hat Linux 9.5.0 with the DISA STIG for Red Hat Enterprise Linux 9 With GUI security profile applied (this will enable SELinux in enforcement mode).

  2. Deploy the Mattermost Server by following the instructions at Install Mattermost on RHEL - Mattermost documentation, being sure to apply the SELinux file context as recommended in the installation instructions.

  3. Attempt to start the service using systemctl start mattermost.service.

Expected behavior
The mattermost.service service should start and remain running, so I can continue on with the rest of the installation instructions.

Observed behavior
When I attempt to start the service, I get the following error:

mattermost.service: Failed to execute /opt/mattermost/bin/mattermost: Operation not permitted
mattermost.service: Failed at step EXEC spawning /opt/mattermost/bin/mattermost: Operation not permitted

This is what my Systemd Unit file looks like:

[Unit]
Description=Mattermost
After=network.target

[Service]
Type=notify
ExectStart=/opt/mattermost/bin/mattermost
TimeoutStartSec=3600
KillMode=mixed
Restart=always
RestartSec=10
WorkingDirectory=/opt/mattermost
User=mattermost
Group=mattermost
LimitNOFILE=49152

[Install]
WantedBy=multi-user.target

Any help is greatly appreciated, thank you.

1 Like
2

Applying the RHEL STIG almost certainly means enabling SELinux (which is a good thing!), and I have a strong feeling that’s what you’re running into… SELinux context labeling. Look in /var/log/audit/audit.log

Chapter 5. Troubleshooting problems related to SELinux | Red Hat Product Documentation should help.

I’m the Mattermost federal pre-sales technical rep :slight_smile: Please feel free to email me at john.oliver@mattermost.com I’d love to know a little more about your environment and use case to see how I can help. I would like to keep the technical discussion around this here in this forum for others to find and refer to, but if there’s anything sensitive you’d like to discuss offline, I’m available.

1 Like
3

Yes sir, it is most certainly SELinux. But it was initially a two-fold problem, I discovered that special characters within the database user’s password had to be properly escaped. So I managed to work through that (using https://urlencoder.org).

I’ve been troubleshooting this by looking at the audit log, as well as journalctl and I’m getting a mixture of “Operation not permitted” and “Permission Denied” errors, depending on what combination of remedies I’ve applied. I’m about to head into the lab now to troubleshoot again, if you’re available, perhaps we can chat. I’ll email you shortly.

I did notice something strange, however. The name of the binary (which is “mattermost”) seems to be truncated whenever it appears in the audit logs. I can post a line from the logs shortly.

PS. I forgot to add that, Mattermost starts fine if I run it directly using /opt/mattermost/bin/mattermost. It’s only failing when starting it as a service using systemctl.

4

What does ls-lX /opt/mattermost/bin/mattermost show? What user are you when you execute the binary directly?

5

Turns out this was an fapolicyd issue.

7

Thanks for chipping in, however, my issue was directly related to FAPolicy. During troubleshooting with @john.oliver, we discovered that stopping the fapolicyd service (with SELinux in enforcing mode and correct file context set in accordance with the installation guide), allowed the mattermost service to start up.

So I applied the FAPolicy settings as recommended in this article, and all is good.

1 Like