Feedback on Collaboration for Mission-Critical Work

In 2024, we are focusing Mattermost’s priorities towards accelerating the world’s mission-critical work, building the most secure, reliable collaboration platform for defense, government, enterprise & critical infrastructure organizations.

As part of this focus, we are looking to update our offerings, and this post is to gather feedback and input from the community most engaged with our work.

Ideas we are discussing here with the community include:

1 - Enterprise: Built for large and advanced organizations seeking a commercial, self-sovereign collaboration platform for mission-critical work. We’re looking to focus the majority of our efforts towards increasing the reliability, security and quality of our infrastructure for use by our core customers. The Professional plan would continue to be available as a pathway to Enterprise plan for organizations with less stringent needs. We are also in the process of determining whether some of the more narrow needs of some customers might be delivered as add-ons or an additional SKU.

2 - Nonprofit: There are some important non-profit and open source organizations that we want to continue to support by providing specially licensed versions of our commercial offerings. For non-profits needing fewer than 1,000 licenses the current in-market non-profit program for self-hosted deployments seems to be working well, where Mattermost has the option of logo and case study right for co-marketing in exchange for highly discounted commercial licenses. For deployments over 1,000 we are working towards a highly discounted cloud-based offering that provides access to our commercial features with hosting from Mattermost, Inc. The cloud-hosted version would have an additional value exchange, where Mattermost would provide pre-released features for the larger non-profit deployments on cloud with an invitation to share feedback on quality and any issues experienced, ahead of roll out to production. The non-profit programs are are separate from our academic programs.

3 - Free: We offer a free plan for teams of less than 50 users seeking a self-hosted team collaboration solution where the most secure way to deploy is to host in private networks behind a firewall or VPN. As the cyber threat landscape continues to accelerate, we are considering taking more prescriptive actions to have Mattermost deployments lacking commercial support to require additional layers of security and to keep deployments closer to the recommended 50 user sizing. The effort that goes into a cyber attack is proportional to the value of the breach, and by encouraging unsupported deployments to stay within the recommended team size of 50, the more we can reduce risk to free deployments. The first step we’re considering is setting a functional limit at 10,000 users on the free version, where additional users cannot be added until other users are deactivated to stay below 10,000 [1].

As we get feedback on the initial change, we’ll look into moving the limit down over time, and supporting qualified non-profit and enterprise organizations in smoothly transitioning to Enterprise and Non-Profit options above without disruption.

As always, we’re very interested in feedback from the community on these ideas. We’d love to hear your feedback on these ideas.

[1] In early November 2023, we had publicly proposed warning messages in product when exceeding recommended user limits in the free version, however initial feedback has suggested functional limits are more clear and effective. Therefore we are pausing warning messages for now due to their complexity, cost and testing requirements vs. the clarity of functional limits:

1 Like

I highly suggest not moving forward with the change to limit open source Mattermost usage by number of users. Such limits are generally frowned upon in the open source world and feels like such a betrayal to the community that has given so much to Mattermost, Inc. Users should be free to decide how much risk they are willing to take on and if they need the additional protections offered in Enterprise Edition. I understand Mattermost, Inc. is trying to increase sales but this isn’t the right way to do it.

A warning banner (Implement Admin Warning Banner for Large Team Edition Installations (500+ Users) to alert of risks of large install · Issue #25291 · mattermost/mattermost · GitHub, Implement End User Warning Banner for Large Team Edition Installations (5000+ Users) regarding application availability and data protection. · Issue #25292 · mattermost/mattermost · GitHub) was only first proposed in November and it appears that has quickly progressed to lockout as of [MM-56318] Global warning banners of user limit for admins by M-ZubairAhmed · Pull Request #25797 · mattermost/mattermost · GitHub - “Upgrade to Mattermost Professional or Mattermost Enterprise to continue using Mattermost.” Such a change in 9.5 would be a breaking change in a minor release.

Ian, Mattermost CEO states in https://github.com/mattermost/mattermost/pull/25797#issuecomment-1884998597 that “we started with safety warning banners” but it appears they actually haven’t been implemented yet? Also regarding “moved to implementing user limits after speaking to our largest deployments … we received surprisingly positive feedback”, where was this feedback collected? There are only a few public mentions all recently that I can find of this proposal and any private interactions are likely largely with existing paying users who would be unaffected and frankly probably benefit from the change despite it going against generally agreed upon on source practices.

My suggestion is, implement the warning banners and not the functional limits. The warning banners will help achieve the stated goals of reducing brand risk and moving users toward paid versions of Mattermost. It’s probably worth noting that in many cases it may drive them to competition like Slack and Microsoft Teams instead though.

Regarding, “It moves larger deployments to either our commercial or nonprofit offerings earlier in their lifecycle” the limits may make sense for new instances and with various warnings along the way so they are not one day caught by surprise when the instance can no longer accept new users. For existing instances such limits just appearing one day will be extremely frustrating and fall on to the shoulders of poor system admins with no purchasing influence. In those cases the options for instances appear to be:

  1. continue running the last version without limits thus leaving it vulnerable to future CVEs,
  2. shutdown, or
  3. have difficult conversations with leadership about upgrading

Mattermost CEO here, @bbodenmiller I appreciate your feedback and thoughtful input on the safe user limits section. One of our core values at Mattermost is “self-awareness” and having dialog with our community is vital to how we work.

You’d referenced the conversation we had on GitHub and I wanted to re-share some of the ideas there to bring community members reading this thread up-to-speed:

What’s most important at Mattermost is that we’re accelerating the world’s mission critical work.

That means earning trust among the defense, government, and critical infrastructure organizations that run us for workflows vital to keeping nations safe, resilient and operational.

As a self-sovereign collaboration platform, one of the key use cases of Mattermost is out-of-band communications for security operations and incident response, and for that market it’s essential that we develop a brand of putting safety and security first.

Part of this is ensuring that any scaled Mattermost deployment–supporting hundreds, thousands or even tens of thousands of users–is running the security, reliability, compliance and automation measures necessary to defend larger deployments and earn the trust of the market.

We started with safety warning banners, and moved to implementing user limits after speaking to our largest deployments, both in nonprofits and in enterprises, we received surprisingly positive feedback.

First, deployments running open source forks of Mattermost can choose to omit the safety limits. By removing the safety limits these organizations explicitly take on security and compliance responsibility, which helps address the brand risk issue. The fundamental tenant of open source is that if you don’t agree with what the project is doing, you can modify it to fit your needs, and we’ve remain committed to that since our founding. Forking a system as complex as Mattermost is a material technical undertaking, so if we’re filtering the large free deployments to those willing to make such a large investment, and reducing the number of large free deployments choosing not to invest, it’s reduced brand risk.

Second, it reduces Mattermost brand risk from failures in free deployments. Poor user experience on large Mattermost deployments can be also be destructive to our brand. As an example, once there’s more than about 3 million posts in the system, search and user experience can become agonizingly slow, and even create work stoppage. The commercial version offers features like Elasticsearch and other capabilities, including onboarding and enterprise support, to correctly address scale milestones, to keep the user experience and workflows fast and productive, and to maintain the Mattermost brand.

I’ve personally been on calls watching an administrator show an 8 second lag from keystroke to input because the enterprise scale features weren’t used, and I sometimes feel like that experience alone enough to require commercial support beyond safe limits.

Third, it moves larger deployments to either our commercial or nonprofit offerings earlier in their lifecycle. Ideally a free Mattermost deployment grows to size where it’s value is clear, and the administrator can then purchase a commercial license at a reasonable deployment size, or apply for a nonprofit license if applicable. Without safe limits, some deployments grow to a size where the initial purchase with a very high user count gets far beyond what they’re able to get through procurement. Then the administrator can become stuck without the enterprise, support and escalation capabilities they need to manage operational risk, and Mattermost doesn’t have the commercial relationship needed to ensure the deployment’s success.

Fourth, this change increases investment in Mattermost for mission critical work. When Mattermost champions in critical infrastructure enterprises are enabled to make the business case for the commercial version at the right time, at a reasonable deployment size, the revenues from those purchases accelerate our ability to invest in building great software and accelerating the success of our champions and end users.

Most importantly, we can increase trust from critical infrastructure enterprises that we’re focused on their success. Having clarity that large deployments of Mattermost are intended for mission critical work on our commercial version at scale (whether that’s DevSecOps, security operations, Command and Control or other use cases in vital organization), protecting our brand from issues impacting free deployments that have gone far beyond safe limits, and having more revenue to invest in our platform are all additive to accelerating the success of mission critical work in the world.

The trade-offs in this direction include a) potentially losing some large deployments of our free version, b) potential brand impact from segments of open source users, customers, partners, internal staff and contributors that don’t agree with the above, c) potential trolling in different forms, rather than constructive dialog and discussion.

We’re still working on the right levels of safety warnings versus limits, and what’s important is that we also work with the largest free deployments to ensure a smooth transition.

There isn’t yet an announcement on these changes, specific limits, or release dates, and we hope to have one as the decisions and designs are finalized.

The major version change to signal a significant change impacting our community has been considered, but given safety limits don’t impact ~99% of free deployments, and that 50 users is the recommended size for free deployments, it could cause more confusion than clarity.

Based on anonymous telemetry fewer than 0.8% of unknown deployments are more than 500 users, and higher numbers, like 10,000, would be closer to 0.1%. This might be a little biased as larger deployments are often within self-sovereign private and air-gapped networks where anonymous telemetry can’t function.

To answer your questions in the context of the other thread:

@bbodenmiller: it appears they actually haven’t been implemented yet?

Correct. Since December we’ve been asking for feedback through this user forum, which gets over 300,000 page views a month, and as the user limit functionality is finalized we’ll have an announcement probably shipping warnings at the 10,000 user level in February and functional limits potentially in March, and then looking at lowering the limits in future releases. The PRs you mentioned are some of the early work in progress.

@bbodenmiller: Also regarding “moved to implementing user limits after speaking to our largest deployments … we received surprisingly positive feedback”, where was this feedback collected? There are only a few public mentions all recently that I can find of this proposal and any private interactions are likely largely with existing paying users who would be unaffected and frankly probably benefit from the change despite it going against generally agreed upon on source practices.

You’re correct. The large deployments were supportive because of the benefits from Mattermost accelerating its ability to invest in the platform and invest in their success with a larger commercial customer base.

The intention of safe limits is to have free Mattermost deployments adopt automation and scale capabilities earlier in their lifecycles, with Mattermost’s customer-facing teams helping them make the business case to stakeholders and to smoothly transition.

Until these changes are made, we’ll continue to have situations where Mattermost champions are in “dilemmas of success”–deployments that have grown the free version far past safe limits, and having higher seat counts, high adoption, and accelerated workflows paradoxically increase the complexity and time needed for procurement.

@bbodenmiller: Regarding, “It moves larger deployments to either our commercial or nonprofit offerings earlier in their lifecycle” the limits may make sense for new instances and with various warnings along the way so they are not one day caught by surprise when the instance can no longer accept new users. For existing instances such limits just appearing one day will be extremely frustrating and fall on to the shoulders of poor system admins with no purchasing influence.

100% agree.

We’ll probably start at a crazy high number like 10,000 users. There’s no admin at that scale that isn’t reading the changelog ahead of an upgrade.

As the change is finalized and rolled out, it’s imperative to Mattermost that we support larger deployments so the user limit change doesn’t disrupt operations.

For deployments that aren’t running their own forks, we want to be ready for Mattermost’s customer-facing teams to provide bridge solutions for free deployments running beyond safe limits, and help champions make the business case to stakeholders to join our commercial community, and ensure the platform performs resiliently at scale, as designed.

Our customer teams have helped Mattermost champions at the world’s most vital organizations win the support of their leadership on Mattermost for self-sovereign collaboration, DevSecOps, out-of-band communications, and Command and Control among other mission critical use cases.

No one wants large free deployments to be degraded or discontinued. At the same time, we are prioritizing our brand, our end users, and accelerating the world’s mission critical work by building a larger commercial customer base across critical infrastructure enterprises.

That said, we can only speak to the deployments who choose to engage with us, through community or commercial channels.

To broaden the feedback, we’re inviting commentary here, on our public forums, which have over 300,000 page views a month. We’re looking for input and dialog from deployments we may or may not know exist.

@bbodenmiller: It’s probably worth noting that in many cases it may drive them to competition like Slack and Microsoft Teams instead though.

Mattermost is designed to integrate with Microsoft Teams and the M365 platform: Mattermost and Microsoft Solutions

By joining Mattermost’s commercial community, deployments will have access to Mattermost integration for Microsoft Teams, along with AI acceleration from the Azure AI platform.

In October 2023, we announced our collaboration with Microsoft’s Defense solutions group to enhance Mattermost’s interoperability with the Microsoft in mission critical environments.

Moreover, when Microsoft announced their roll-out of Azure AI for classified workloads in the U.S. Department of Defense, they’ve highlighted Mattermost as a key ecosystem partner: 🤯 Microsoft 2024 roadmap is transformative for US Government. Meet Mattermost and Mobius Logic. | Ian Tien posted on the topic | LinkedIn

Here’s Microsoft’s announcement:

Microsoft: Another example is a new collaboration with Mattermost and MobiusLogic. Together, we are working to deliver improved collaboration and workflow solutions for national security customers built on Azure’s secure cloud and leveraging Azure AI capabilities. The solution stack is built on the Mattermost OpenOps AI experience platform connected across Microsoft’s core platform services including Azure AI Services, Microsoft Entra ID, and Microsoft Teams. The hub offers a highly interoperable collaboration experience with advanced levels of customization, integration, AI-acceleration and deployment across private cloud, public cloud and air-gapped networks.

Mattermost is often deployed as a supplement to Microsoft Teams.

By connecting Microsoft Teams to Mattermost, or running Mattermost along side, central IT solves a host of problems, from needing a self-sovereign platform to address regulations and data control complexity (e.g. including addressing Schrems II in Europe), to out-of-band communication for security organizations that might need to run incident response and mitigations concerning the M365 platform, to DevSecOps organizations that need to focus on technical workflows and integrating to rapidly and dynamically changing toolchains and in-house systems.

Again, @bbodenmiller, really appreciate your input and feedback. Thanks again for asking publicly so we can share answers with our whole community.

Very open to other feedback, per the title of the thread,

In my opinion this is way too fast of an escalation for such an impactful breaking change. Many large instances will be on a 3 or 6 month upgrade cycle and thus will not even notice the in-product warnings and have a chance to give feedback before the limit is implemented. Will the blog get a post asking for feedback? In conversations with folks I’ve heard of more people watching the blog than the forum. Would also be good to add warnings of incoming change to Pricing | Mattermost which in multiple places states unlimited for free version.

Furthermore, must organizations budget on an annual basis so it’s unlikely they could make a purchase in just a few months.

For what it’s worth, I’ve seen the open source version run much better than this when running on appropriately sized hardware.

Would love to see above addressed. I haven’t seen precedents for such limits in open source software. I know GitLab for example is against such limits, only locking entire features behind the paywall.

I believe most large instances of Mattermost are either running to avoid the need to buy Slack or Microsoft Teams, or were setup before those SaaS solutions were viable options for various reasons (e.g. data sovereignty requirements). In the case of Mattermost implementing limits, I suspect many will succumb to central ITs desire to move to either Slack or Microsoft Teams (both available at lower price points) leaving Mattermost for only extremely niche use cases which would likely be under the proposed user limits & thus still may not result in a sale and thus the limits not achieving the stated goals.

A better option may be to allow different tiers per user. Imagine an instance with 10,000 users of which 1,000 have the business justification for paid features and the other 9,000 do not. Buying 1,000 seats is a lot easier purchase than 10,000. Over time that could increase from 1,000 out of 10,000 to 2,000 out of 10,000 and so on which is a lot easier than going from 1,000 to 10,000 overnight. I understand some features are at an instance level like HA so the unpaid users would get those for free. Many instance level features could still be flagged behind the paywall (e.g. only paid users get elasticsearch search experience).

For the sake of achieving your goals but not making it painful for those of us who maintain an open source fork: Please consider adding a build-time setting/flag for this so one can set an env var during build to change the hard-coded 10000 limit to something higher or simple have a boolean flag to disable it.

That way you have the 10000 users limit default in your official builds, and maybe even in default Open Source builds (assuming you want this to be enabled by default by distro packages who build from source, since that may be a valid source of future customers after all). But by adding a build flag to allow the maintainer to consciously - yet still easily and without the annoyance of git merge/rebase conflicts - opt-out from the user limit you make their life easier. And I think this is also beneficial for you because some of those people who run such large open-source instances may actually give you valuable feedback!

(PS: I think you know who I am and which instance I run @it33_mattermost considering we briefly met mid-november - just to indicate that the ahove comment is actually from someone who maintains a large instance and not from some random person in the interwebs ;)).

Thanks @ThiefMaster, certainly appreciate your partnership, and your feedback, and getting a tour of your facilities last year. It was quite incredible to see the impact Mattermost has had in your world, and it was super fun dropping by while I was in town.

There’s no intention to make opt-ing out of safe limits unnecessarily complex for forks.

For Feb we’re shipping a warning message for safe limits at 10,000 users to the administrator, and we can look into shipping something configurable in the build process when we implement the actual limits in March.

What’s important to us is that organizations running Mattermost at scale without our commercial or non-profit builds are doing it responsibly, and responsibly managing an open source Mattermost fork is a clear signal of capability.

For others joining the thread here, I’ll just quote from above for context on the 10,000 user safety limit as it applies to forks:

I empathize with any Mattermost champion impacted by Team Edition safety limits at 10,000 users and feeling “betrayed”. I really feel bad about this, because it’s unfair.

While Team Edition is recommended for up to 50 users, we didn’t have safety limits previously, so adding safety limits at 10,000 users, even though that number is extraordinarily high, it does impact a handful of deployments who have actually done an enormous service for the Mattermost community by driving adoption to such gargantuan levels.

We owe you an explanation. We owe you a path forward. I’m trying to be thorough in this forum response to provide both, with context.

My hope is when this is in the rear view mirror that it might be an opportunity for a different kind of success in the careers of our champions. The challenge we see with very large open source deployments is they can wobble after the founding admin moves on. Being the single point of failure of a massive, mission critical system is an enormous and pressurizing responsibility for many.

My wish is that we can support this small number of elite and heroic champions and flow towards a shared vision of enduring success and buy-in for what they’ve achieved.

Part of this is building a relationship with Mattermost’s customer organization and progressing on a healthy path towards the enterprise-level infrastructure, support and investment to enable Mattermost deployments endure and flourish in the long term.

It sounds like the question above is asking about reasons, principles and impact, so I’ll try to cover that with fidelity and with examples, and forward pathing, while also providing context for the broader readership:

What is Mattermost Team Edition?

“Mattermost Team Edition” is a Linux binary compiled by our commercial organization from our open source code base and provided under an MIT license.

It’s named “Team Edition” because it was intended for a single team. In a team, everyone knows everyone. They trust each other. They work together collaboratively. Team Edition has a recommended size of 25-50, because beyond that size, we’re not really talking about a single team.

When an administrator asks for Microsoft Entra ID integration for the open source Mattermost Team Edition so users who don’t know each other can be identified from attributes automatically pulled into the system, it’s clear that it’s not a single team use case, and it should be an enterprise capability in our commercial version.

That’s the simplicity that comes from “Team Edition” naming.

Even with a max recommended size of 50, it’s probably fine going to a couple of hundred users without material issues, but 10,000 is kind of bananas. We are putting in “bananas-level” safety limits.

Why does Mattermost Team Edition exist?

The purpose of Mattermost Team Edition is to drive brand awareness of Mattermost in critical infrastructure enterprises by enabling hands-on deployment and usage.

We provide it under MIT license, which a lot of developers are approved to use. It works with PostgreSQL, which is a common and standard open source database, enabling the administrator to control 100% of the data produced on the system.

Still, it’s very hard to get into many critical infrastructure enterprises, because a lot of them actually ban open source by default.

For those organizations to use Mattermost Team Edition, there’s often a series of security reviews and analysis on our open source code base, including automated scanning, and sometimes multiple groups in parallel need to independently sign-off.

These reviews typically happen without our knowledge or interaction. People are reviewing our source code, they’re going over our website, they’re researching me, our engineering and security leadership, the public information shared by our contributors, and they’re assessing the current and future risk of bringing our open source code into mission critical environments.

We invest a lot in getting Team Edition on the “approved” list at many of these organizations, and in building the brand of Mattermost in trust and safety.

This includes managing our Mattermost Responsible Disclosure Policy to confidentially receive, review and address reported vulnerabilities, joining the Common Vulnerability and Exposures program funded by the U.S. Department of Homeland Security and becoming a CVE Numbering Authority (CNA) to enable public tracking of vulnerabilities and remediations, funding and operating the Mattermost Bug Bounty program, building trust with the global security community from leading coordinated security remediation and disclosure efforts with platform technologies such as Golang, to joining the World Economic Forum Cybersecurity Community to accelerate global resilience. Moreover, as a mission-critical end user application, when we’re not able to effectively automate QA, we run thousands of QA tests manually with every monthly Mattermost release.

In the context of all we’re investing in brand, introducing safety limits at 10,000 users reduces brand risk from user experience and resilience issues on the largest free deployments of Team Edition that are running without the requisite automation and scale capabilities, and without commercial support.

How do safety limits clarify and accelerate Mattermost’s open source work?

As a commercial open source company our intention is to accelerate investment in our platform and open source work over time, by growing our commercial customer base and expanding our offerings.

Part of this work is clarifying the full benefit of our commercial subscriptions, offerings and operations relative to our open source work. There’s three pieces here:

1 - The Mattermost open source project

Our open source code base is publicly available.

For anyone who’s contributed to our open source repositories, I’d welcome input and feedback on this thread, this section, or any section.

In my mind, open source software is available for anyone to copy, modify, compile and use.

The value exchange in open source communities is typically to have open source contributions be available in the open source code base, and having a process by which contributions are publicly acknowledged and traceable through the contribution process, which we do via GitHub.

I think of different limits set as constants or variables as common in software, including open source software. Adding the capability to define a maximum safe limit for user count in the code base doesn’t seem controversial to me, but the point of this thread is to hear feedback, and uncover blind spots so input is very welcome here.

I understand that if we compile a specific version of Mattermost’s open source codebase with a safety limit set at a number some members of the community don’t agree with–in this case 10,000 users for a compiled version that recommends 50–it will likely be frowned on by anyone impacted, because it can make their work more difficult in the near term.

At the same time, I don’t know if adding a setting in a specific compiled version of an open source code base would be seen as an infraction on open source principles, since you anyone can compile their own version with any settings they choose.

2 - Mattermost Team Edition Linux binary

Team Edition is a compiled from our open source software by our commercial team, and provided with an MIT license and an array of trust and safety and initiatives, for the purpose of expanding Mattermost brand awareness in critical infrastructure enterprises through hands-on usage.

Our priority is accelerating the world’s mission critical work, and Team Edition is our brand ambassador for achieving that purpose.

The Team Edition experience is optimized for a single team, around 25-50 users. While it’s probably okay to scale to maybe a couple hundred or even a few hundred users on Team Edition, our intention is to have Mattermost champions start considering Mattermost’s commercial offerings for automation, scale and enterprise support capabilities around the time deployments grow past the recommended size.

When hundreds of users in a critical infrastructure enterprise have adopted the free version of Mattermost for workflow, it’s typically straight forward for Mattermost champions to collaborate with our customer-facing teams in developing a business case for IT, finance, procurement audiences and flowing through the organization’s infrastructure planning and budgeting processes.

There’s a host of spend categories where Mattermost excels and a straight forward mapping to budget categories in critical infrastructure enterprises, including security operations/out-of-band communications, engineering/DevSecOps infrastructure, compliance/self-sovereign collaboration, mission systems/Command and Control, digital workplace/Microsoft Teams enhancement, among other options.

If Mattermost is fulfilling its purpose, and accelerating mission critical work in critical infrastructure enterprises, it’s not uncommon to find department-level budget for a few hundred users after making a clear business case.

In addition to scale and automation, our commercial offerings provide enterprise support, federation, administration and compliance, resilient Kubernetes-based operations with high availability and fault tolerance deployment, and a host of capabilities for serving tens of thousands of users over time.

Most importantly, the commercial offerings provide a relationship between Mattermost and it’s largest scale deployments, which have the greatest impact on our brand.

When a Team Edition deployment grows far past its intended size and without the capabilities designed for that level of scale, we have seen some rocky end user experiences, including extraordinarily long page load times that make systems nearly unusable, extended lags between keyboard input and display, freezing of displays, in addition to foundational deployment issues.

These examples of degradation often happen when the original Mattermost champion has moved to a different role or been promoted (most Mattermost admins are quite strong, and high performers), and a new owner needs to manage the system without training, support and without the requisite product capabilities.

Without an experienced admin, and without commercial support and without enterprise capabilities there’s very high risk of Team Edition either being a zombie instance with degraded user experience, or being shutdown completely.

With a commercial relationship the legacy, value delivery and brand of Mattermost deployments endures and expands. Mattermost customer success teams can support transitions to new admins, guiding them with training on the system, understanding system capabilities, continually improving operations and value and making the business case for continued and extended support from IT, finance and procurement stakeholders.

Part of implementing safety limits in Team Edition is to draw attention from all Mattermost champions running free deployments to understand that there’s a healthy path to having their Mattermost investments endure long past their tenure, to make scale, administration and automation dramatically easier, to influence a meaningful, impactful commercial investment from their organization, and to accelerate the resilience and success of their enterprise.

That’s kind of why this forum response is so long and detailed. :slightly_smiling_face:

Adding safely limits at 10,000 users to the compiled Linux binary we call Team Edition doesn’t really impact anyone who compiles from our open source code, since it’s straight forward to change (see top of this post).

It does impact Team Edition deployments over 10,000 users, and Mattermost’s customer-facing teams will be fully supporting those deployments to avoid any operational impact in the journey towards commercial capabilities.

A key objective of this change is to enable Mattermost champions who have invested so much into their deployments to have their achievements endure, expand and leave a legacy of success in their organizations and in their careers.

3 - Mattermost Enterprise Edition

The third component to discuss here is the commercial version of Mattermost, which provides the commercial relationship, support, account management and broad array of enterprise product capabilities to not only scale but to flourish.

If you’re a Mattermost champion in a critical infrastructure enterprise who’s got a few hundred users on Team Edition, and you’re seeing the system used for mission critical work, and thinking about the future scalability and legacy of the system, it might be the right time to consider starting on the path to joining our commercial community.

You can review our enterprise capabilities below:

https://docs.mattermost.com/about/editions-and-offerings.html#mattermost-enterprise

The section above includes guidance on how to contact Mattermost’s customer-facing teams as well.

Ideas and input are very welcome. I need to be a little careful in replying here as our customer-facing teams are really the ones that work through these kind of discussions.

Right now the safety limit is at 10,000, so for any deployment where that limit is an issue, I think the first step would be to understand the value of Mattermost for all the current users, and which organizations would budget to enable those users, if any.

Once the data is in, it might be easier to consider different options.

Again, thanks to both @bbodenmiller and @ThiefMaster for the feedback here.

This is an important issue for our largest free deployments, which is why we’ve opened the discussion several months before the changes are planned to be shipped, and have asked for feedback on the change so we can address the top issues when we finalize and announce.

@bbodenmiller, I hit the character limit on the forum for a reply, so trying to answer your other questions here. I really appreciate your feedback, which was requested on this thread, and I’ll be trying to cover down on your points.

This thread is getting long so I might intentionally repeat/adapt some of the context from our previous discussions to err on the side of clarity.

There’s some risk of me getting called out for sounding like I’m repeating what I said before, but repeating some statements in a long thread is how I’m hoping to de-risk people missing part of what I’m trying to share.

The intention of safe limits is to have free Mattermost deployments adopt automation and scale capabilities earlier in their lifecycles, with Mattermost’s customer-facing teams supporting champions in making the business case to their stakeholders and to smoothly transition.

Until these changes are made, we’ll continue to have situations where Mattermost champions are in “dilemmas of success”–deployments that have grown the free version far past safe limits, and having higher seat counts, high adoption, and accelerated workflows paradoxically increase the complexity and time needed for procurement.

We thought of 10,000 users as an “insanely high” number for the starting point of safe limits, for a free product who’s recommended size is 50. The intention was to have impact limited to a very, very small number of deployments on the planet, which we could work with directly to ensure they’re continuing to smoothly operate.

In my mind, in running 10,000 users without enabling automation and scale capabilities, the Mattermost brand has the most extreme risk exposure: a) more things can go wrong, b) more end users impacted, c) longer time to mitigate the issues with procurement and implementation changes, d) geometrically greater brand risk given a * b * c.

So at the 10,000 user starting point, our thinking is we have a geometrically smaller number of deployments impacted (we estimate ~0.8% of free servers have more than 500 users, and 10,000 is nearly zero), and a geometrically greater risk.

That said, I’d like to ask @bbodenmiller a clarifying question–with the caveat that this isn’t something we’re necessarily committing to doing, as there are many stakeholders involved in this–but for you or anyone in the community:

If we shipped user limit warnings at 10,000 in Feb, for admin only, and functional user limits were at 25,000 for March, would anyone consider that a breaking change?

Whether it’s 10,000 or 25,000 users, if the upgrade cycle is 3-6 months, and the functional limits to the free Team Edition ship in March, then it sounds like it’s around 5-8 months from now until when users above 10,000-25,000 need to be deactivated from the system, or a bridge solution to a proof-of-value environment can be deployed, working with the Mattermost customer team.

Did I get that right?

I can’t personally conceive of running at 25,000 users on Team Edition, without scale and automation features enabled, without an incredible number of things breaking and being destructive to the brand.

I’m trying to understand if the issue is user limits in themselves, or certain free instances being impacted.

Regardless of the answer, it’s imperative that our customer-facing teams are connecting with our largest free deployments–especially ones we don’t know about yet–so upcoming changes don’t impact operations for any instance.

Side note: I think there’s going to be a safe limit user limit setting so high that effectively no one is impacted, and hence it’s not a breaking change. The analogy here is SpaceX Starship. My understanding is when getting approval to fly an experimental 100 ton rocket over the ocean, there was a concern that the falling debris from a failure might hit a whale. When the math was done, the surface area of whales relative to the surface area of the ocean was effectively zero. So it’s kind of the same here, there’s some super high limit we can use as a starting point that won’t hit any actual instance, and therefore wouldn’t be a breaking change.

We’re planning to do a blog post based on what’s shared at the top of this thread (Feedback on Collaboration for Mission-Critical Work) which is about accelerating mission critical work in the world.

It’ll discuss our focus on critical infrastructure organizations with our enterprise offerings, also discuss our nonprofit self-managed license for communities below 1000 users. We’ll also be potentially announcing a new cloud offering for nonprofits over 1000 users, very steeply discounted, with a value exchange where they may receive earlier versions of new improvements to share their feedback ahead of rolling out to broader audiences.

We’ll be discussing the changes in our free offering as well, with recommendations on securing deployment configurations across private networks and behind VPNs to provide a layered security foundation for the platform. We’ll also be sharing about safe limits rolling out, starting at 10,000.

There’s not a mechanism on the blog to provide feedback on the website, but we can look into linking to the Mattermost forums, or this thread specifically.

Interestingly the reason we had “unlimited” on the pricing page was that we used to have a cloud freemium version limited to 10 free users.

We iterated on the 10 user limit in free, and put “unlimited” for users in, but limits in other areas.

Then the cloud freemium version was discontinued, so there’s only the commercial cloud offerings now, but I think the pricing page needs to clean up the artifacts, because all versions–including paid versions–should have safe limits depending on their configuration.

Appreciate the feedback here, we should remove the out-of-date mentions.

100% agree.

I’m personally committed to ensure we successfully bridge anyone impacted by this change.

It’ll be unsurprising to our customer-facing teams that transitions can take more than a few months.

Sometimes it feels like the larger the purchase, the longer it takes, even when you’ve got a ton of end users using the system, and more users at the doors asking for access.

Often it’s engineers, technical teams and operators driving Mattermost adoption. These are the folks at the core of every critical infrastructure company, and we’re focused on accelerating their most vital workflows.

When Mattermost transitions from free to commercial early, it’s an easier path because it’s within discretionary budgets of technical leadership, there’s validation from tech teams that their workflows are faster, that their transparency and focus is higher, and error rates are reduced–because Mattermost is built for for operators and technologists.

Supporting Mattermost champions in making the business case, and winning the support of stakeholders is what our customer-facing function is built to do.

This includes all the support to keep things operational including figuring out bridge solutions, proper sizing, architecture selection, roll out planning, trial licenses, migration acceleration, and so forth.

In enterprises, when a free Mattermost deployment hits 100-500 users, IT often declares it a “successful pilot”. Then either the champion or procurement contacts Mattermost, or one of our resellers, and works with a mix of our online documentation, our system integrator community, and Mattermost support and technical account management teams on upgrading to one of our commercial versions, accelerating automation and scale, and to smoothly roll out to internal users.

There are larger transitions in the 1000s that have been a little more work. For a 10,000 user transition from free to commercial it might be a mix of pilot-to-production and upgrade-for-scale flows. I think the best outcome would be landing on the Kubernetes operator in a scaled, high availability environment so all the future management would flow smoothly.

That’s very good to hear. We’ve seen some really painful ones.

At the same time, no one really calls us when everything is working well, so our data isn’t super normalized.

One of the reasons of setting safely limits at 10,000 users on the Team Edition was that we didn’t expect IT departments to start considering a massive Microsoft Teams or Slack purchase, or exiting Mattermost all together, when the limit is astronomically high.

A few thoughts here:

Developing a free-to-paid business case at 10,000 users.

If I was a CIO and someone told me there was a piece of open source infrastructure that had a new limit of 10,000 users, coming down over time, I imagine I’d be like “Okay. Sounds like we get 10,000 free users. Figure out who to remove so we can stay under the limit, and let me know when they announce limits coming down, and we can deal with it then.”

What we intend for safety limits to trigger, both at 10,000 users and in future at lower levels, is an assessment of Mattermost’s value among its user base.

This happens regularly in our paid deployments for initial purchase, for license expansion and for license renewals. There’s a process to speak to all the organizations using Mattermost to understand:

a) why they want it,

b) what’s the benefit for them of continuing to have it, and

c) whether the organization’s would use their own budget to buy seats.

Some orgs may or may not have their own IT budgets, so sometimes it’s a “what if” assessment.

Through this process, the business value of Mattermost seats is established and the data can be provided to Central IT for review and budgeting.

In some enterprises, Central IT works through a charge-back system to group buy licenses for all the organizations that want them, and then get reimbursed by organizations using the shared services. In other enterprises, Central IT might just allocate the purchase from a shared services budget, or a departmental budget (e.g. product and engineering), or there could be a mix.

Meanwhile, it’s a priority for Mattermost’s customer-facing teams to support Mattermost champions in working with their organizations through this process, ensuring there’s no operational disruption, as well as sharing patterns, practices and materials to speed success in the commercial journey.

Why buy Mattermost, when Microsoft Teams and Slack are available?

I think Microsoft Teams has an estimated 300 million users right now, so I can certainly see the perspective that everything else is “niche”, perhaps Slack included.

What I can share publicly is what’s available on our website, which is the use of large Mattermost deployments for mission critical work.

This includes the U.S. Department of Defense using Mattermost for operational ChatOps in complex and real world environments, and the use of Mattermost in one of its largest-ever readiness exercises for providing airlift, aerial refueling, aeromedical evacuation, and humanitarian and disaster assistance.

We also enable critical infrastructure enterprises internationally, including utilities like RTE who manages France’s national power grid.

So my perspective is slightly different.

I think of Mattermost’s focus on accelerating the world’s mission critical work as an important and large market, and a meaningful mission for our company and our community.

When the work of an organization is vital to the safety of a society, there’s an elevated need for resilience, and for platforms dedicated to the success of operators and their technical teams, which can adapt to their specific needs.

This is the business case made over and over again within our commercial customer community.

Resilience in Microsoft Teams environments

In the context of resiliency, Mattermost’s ability to supplement Microsoft Teams is an essential service for many customers who run both.

My co-founder and I are former Microsoft engineers, that worked out of Redmond, and we have enormous respect for many of the security professionals that work there. At the same time, the cyber pressure Microsoft faces is extraordinary and increasing, due to the growing magnitude of confidential data they are protecting, advancements in offensive cyber capabilities and in the sheer ferocity in the newest generations of attacks.

It’s difficult for many critical infrastructure enterprises to get comfortable storing 100% of their confidential communications in Microsoft Teams as a multi-tenant shared service. Especially when it’s operating on a shared code base on public cloud with known points of ingress and egress, and uses availability keys in its encryption protocol.

I’ve talked to some security professionals who share that for safety reasons they need to operate under the assumption that Microsoft Teams is breached at all times, and any incident response or discussion of confidential vulnerabilities data needs to happen in an “out-of-band” system.

Part of this is the principle that security operations should be separated from the environments they’re protecting. So if security professionals are protecting a network, they should be on a separate network. If they’re protecting Microsoft Teams, defending against threats and responding to breaches or new vulnerabilities potentially in M365, they should be on a platform out-of-band from Microsoft Teams.

The other part of this is the principle that the effort that goes into a breach is proportional the economic value of the breach. At 300 million users, the economic value of breaching Microsoft Teams is astronomically high for attackers, compared to the value of breaching a self-sovereign Mattermost system used by only a single target.

Also this doesn’t apply just to Microsoft Teams, if you have any primary platform for collaboration that your security operations is tasked to defend they should have an out-of-band platform for security operations.

As a counter example, when you read through the public SolarWinds disclosures, it seems clear the security organization didn’t use out-of-band communication, so when their primary collaboration system was breached without their knowledge, information about vulnerabilities in their systems and in their customer base, as well as their response plans were flowing to attackers, compounding the problem and creating a cycle where recovery was somewhat impossible.

Beyond the security perspective, for resiliency in mission critical operations, out-of-band scenarios also apply to platform engineering, SRE, DevOps, and DevSecOps scenarios when organizations want control of their own uptime SLAs.

From this context, for any critical infrastructure enterprise running Microsoft Teams or any other primary collaboration platform at scale, there is an opportunity for Mattermost to provide out-of-band support for resilience in addition to workflow acceleration.

Focus and Adaptability to mission critical workflows

Mattermost provides operators, technologists and developers an environment focused on accelerating their vital workflows, increasing their visibility, and reducing error rates and blind spots in the delivery of mission critical work.

For technologists and DevSecOps practioners, Mattermost’s ability to integrate with systems like GitLab, to be customized, to integrate with custom tooling, in-house systems, custom security and compliance requirements, as well as connecting into DevSecOps infrastructure inside and outside the Microsoft portfolio all contribute to providing the focus needed to accelerate technical and operational success.

For operators in mission critical and high compliance environments, Mattermost’s open source code base and self-sovereign design lets organizations adapt Mattermost to nearly any purpose–deployment in private and air gapped networks, integration with custom compliance tools, customized federated communications across security boundaries, deep AI automation and a host of scenarios and capabilities that centralized SaaS platforms would struggle to provide.

In contrast, general collaboration tools like Microsoft Teams are designed to serve “everyone”, and there can be a level of information overload for end users, a level of request overload to Central IT for supporting specialization, and a level of system limitation that makes mission critical work significantly more difficult to complete.

Without the right tool for the job, without the right support for the vital work of operational and technical organizations, the risk of blind spots, errors and delivery delays can elevate. This can lead to risks in retaining the strongest engineers, and recruiting new stars. Unaddressed, these risks all together can snowball towards compromising operational resiliency and success for the enterprise itself.

Mattermost is built to address these risks, both independently and as a supplement to Microsoft Teams through our interoperability investments: Mattermost and Microsoft Solutions

Circling back to your question on Slack, we don’t see them that often in critical infrastructure enterprises, and my understanding is our value propositions are quite difference.

While Slack in its startup days served developers, after its acquisition by Salesforce its user interface was re-designed and it seems its long term direction has shifted to sales and marketing integration, with a new Slack CEO publicly announcing in 2023: “My vision is that we become the core engagement layer for all of Salesforce, for how people engage with Salesforce. There should be no starting point besides Slack.” Also, our understanding is that to become a Slack customer, CIOs need to approve Slack’s policy of deleting all customer IP stored in Slack when the customer downgrades their subscription tier.

Mattermost take a different approach, with a focus on critical infrastructure enterprises and accelerating the success of operators, technologists and developers in their mission critical work, and offering self-sovereign deployment and full control of private data and IP.

@bbodenmiller thanks again for the feedback, which is what this thread is for. Very open to more input from you or any reader.

Thanks for the detailed response @it33_mattermost - it’s a lot to consume.

IMO it’s still a breaking change even if it affects minimal instances.

Is the limit based on monthly active users, concurrent users or “activated” users? Long running instances could have a lot of “activated” users with much much less monthly active or concurrent users.

I believe it’s register users that aren’t deactivated. This can probably be confirmed in the code as it rolls out.

Would there be a difference if the changes impacted zero instances?

I’m curious your thoughts on how SpaceX Starship was approved to launch:

Easier said than done without robust user management features not present in Team Edition. With the limit presumably based on activated users rather than monthly active users, this means once a user is deactivated they cannot self-serve return without admin intervention creating a host of problems. Sure it may drive license sales but it may just drive people away.

Maybe but with the implementation timeline discussed here (limit starting in March release) it seems like the limit will be in place before some admins are even seeing the warning let alone get ahold of Mattermost, Inc. to discuss options. It is not uncommon for admins to skip releases.

Environmental Law is outside my area of knowledge or expertise :slight_smile:

@it33_mattermost this is an interesting explanation of Team Edition that I have not seen (or simply missed) before. Pricing | Mattermost hints at it, though as previously discussed “Unlimited Teams” confuses the point. Mattermost editions and plans — Mattermost documentation doesn’t really seem to communicate the point that it’s designed for a single team until you get to plans which at skim I would have assumed would only apply to Enterprise Edition.

I agree the name Team Edition given the explanation makes sense but I’ve always just seen it as Mattermost open source and Mattermost paid edition - I’ve only used it via GitLab Omnibus package which doesn’t really seem to call out the intended use case or edition - GitLab Mattermost | GitLab. Interestingly, upon looking now I see the download for Team Edition is even buried away quite a bit and not mentioned much. Also unclear how or if it can be used on Kubernetes.

If Team Edition is made for a single team it seems it may have made sense to restrict the multiple-Teams capability to Enterprise Edition when introduced. I would highly recommend not making that change now. This thought exercise has helped clarify what bothers me most about the 10,000 user limit: it’s moving/restricting a capability currently in the open source version to the enterprise version. Removing/restricting capabilities in open source tier in favor of paid tier is something that a number of companies backing open source products have vowed not to do, like GitLab (stewardship promises):

When a feature is open source we won’t move that feature to a paid tier.

I had hoped Mattermost, Inc. operated under a similar promise given how similarly the two companies operate.

Now if we’re talking implementing an overridable safety limit that could be a bit different conversation (same story if it were just an annoying safety warning to admins). GitLab promises “the open source codebase will not contain any artificial limits” but they do include sane default limits which can be overrode (though not always easily) if admin believes/sets up infrastructure to handle it. If Mattermost, Inc. were to allow the safety limit to be changed or override (without re-compiling the code base), I don’t think this is such a controversial change (whether it’s breaking or not is still up for debate). Without such override I see the change as ultimately commercially motivated, and not in line with the promises other companies have made in regards to the open source software they maintain.

@ThiefMaster we’re adding the change you suggested - using a built-time variable to configure limits, making it easier for fork maintainers. You can find the PR for the same at Updated the user limit to make it easier for community forks by harshilsharma63 · Pull Request #26105 · mattermost/mattermost · GitHub

Thanks!

1 Like