In 2024, we are focusing Mattermost’s priorities towards accelerating the world’s mission-critical work, building the most secure, reliable collaboration platform for defense, government, enterprise & critical infrastructure organizations.
As part of this focus, we are looking to update our offerings, and this post is to gather feedback and input from the community most engaged with our work.
Ideas we are discussing here with the community include:
1 - Enterprise: Built for large and advanced organizations seeking a commercial, self-sovereign collaboration platform for mission-critical work. We’re looking to focus the majority of our efforts towards increasing the reliability, security and quality of our infrastructure for use by our core customers. The Professional plan would continue to be available as a pathway to Enterprise plan for organizations with less stringent needs. We are also in the process of determining whether some of the more narrow needs of some customers might be delivered as add-ons or an additional SKU.
2 - Nonprofit: There are some important non-profit and open source organizations that we want to continue to support by providing specially licensed versions of our commercial offerings. For non-profits needing fewer than 1,000 licenses the current in-market non-profit program for self-hosted deployments seems to be working well, where Mattermost has the option of logo and case study right for co-marketing in exchange for highly discounted commercial licenses. For deployments over 1,000 we are working towards a highly discounted cloud-based offering that provides access to our commercial features with hosting from Mattermost, Inc. The cloud-hosted version would have an additional value exchange, where Mattermost would provide pre-released features for the larger non-profit deployments on cloud with an invitation to share feedback on quality and any issues experienced, ahead of roll out to production. The non-profit programs are are separate from our academic programs.
3 - Free: We offer a free plan for teams of less than 50 users seeking a self-hosted team collaboration solution where the most secure way to deploy is to host in private networks behind a firewall or VPN. As the cyber threat landscape continues to accelerate, we are considering taking more prescriptive actions to have Mattermost deployments lacking commercial support to require additional layers of security and to keep deployments closer to the recommended 50 user sizing. The effort that goes into a cyber attack is proportional to the value of the breach, and by encouraging unsupported deployments to stay within the recommended team size of 50, the more we can reduce risk to free deployments. The first step we’re considering is setting a functional limit at 10,000 users on the free version, where additional users cannot be added until other users are deactivated to stay below 10,000 [1].
As we get feedback on the initial change, we’ll look into moving the limit down over time, and supporting qualified non-profit and enterprise organizations in smoothly transitioning to Enterprise and Non-Profit options above without disruption.
As always, we’re very interested in feedback from the community on these ideas. We’d love to hear your feedback on these ideas.
[1] In early November 2023, we had publicly proposed warning messages in product when exceeding recommended user limits in the free version, however initial feedback has suggested functional limits are more clear and effective. Therefore we are pausing warning messages for now due to their complexity, cost and testing requirements vs. the clarity of functional limits:
I highly suggest not moving forward with the change to limit open source Mattermost usage by number of users. Such limits are generally frowned upon in the open source world and feels like such a betrayal to the community that has given so much to Mattermost, Inc. Users should be free to decide how much risk they are willing to take on and if they need the additional protections offered in Enterprise Edition. I understand Mattermost, Inc. is trying to increase sales but this isn’t the right way to do it.
Ian, Mattermost CEO states in https://github.com/mattermost/mattermost/pull/25797#issuecomment-1884998597 that “we started with safety warning banners” but it appears they actually haven’t been implemented yet? Also regarding “moved to implementing user limits after speaking to our largest deployments … we received surprisingly positive feedback”, where was this feedback collected? There are only a few public mentions all recently that I can find of this proposal and any private interactions are likely largely with existing paying users who would be unaffected and frankly probably benefit from the change despite it going against generally agreed upon on source practices.
My suggestion is, implement the warning banners and not the functional limits. The warning banners will help achieve the stated goals of reducing brand risk and moving users toward paid versions of Mattermost. It’s probably worth noting that in many cases it may drive them to competition like Slack and Microsoft Teams instead though.
Regarding, “It moves larger deployments to either our commercial or nonprofit offerings earlier in their lifecycle” the limits may make sense for new instances and with various warnings along the way so they are not one day caught by surprise when the instance can no longer accept new users. For existing instances such limits just appearing one day will be extremely frustrating and fall on to the shoulders of poor system admins with no purchasing influence. In those cases the options for instances appear to be:
continue running the last version without limits thus leaving it vulnerable to future CVEs,
shutdown, or
have difficult conversations with leadership about upgrading
Mattermost CEO here, @bbodenmiller I appreciate your feedback and thoughtful input on the safe user limits section. One of our core values at Mattermost is “self-awareness” and having dialog with our community is vital to how we work.
You’d referenced the conversation we had on GitHub and I wanted to re-share some of the ideas there to bring community members reading this thread up-to-speed:
What’s most important at Mattermost is that we’re accelerating the world’s mission critical work.
That means earning trust among the defense, government, and critical infrastructure organizations that run us for workflows vital to keeping nations safe, resilient and operational.
As a self-sovereign collaboration platform, one of the key use cases of Mattermost is out-of-band communications for security operations and incident response, and for that market it’s essential that we develop a brand of putting safety and security first.
Part of this is ensuring that any scaled Mattermost deployment–supporting hundreds, thousands or even tens of thousands of users–is running the security, reliability, compliance and automation measures necessary to defend larger deployments and earn the trust of the market.
We started with safety warning banners, and moved to implementing user limits after speaking to our largest deployments, both in nonprofits and in enterprises, we received surprisingly positive feedback.
First, deployments running open source forks of Mattermost can choose to omit the safety limits. By removing the safety limits these organizations explicitly take on security and compliance responsibility, which helps address the brand risk issue. The fundamental tenant of open source is that if you don’t agree with what the project is doing, you can modify it to fit your needs, and we’ve remain committed to that since our founding. Forking a system as complex as Mattermost is a material technical undertaking, so if we’re filtering the large free deployments to those willing to make such a large investment, and reducing the number of large free deployments choosing not to invest, it’s reduced brand risk.
Second, it reduces Mattermost brand risk from failures in free deployments. Poor user experience on large Mattermost deployments can be also be destructive to our brand. As an example, once there’s more than about 3 million posts in the system, search and user experience can become agonizingly slow, and even create work stoppage. The commercial version offers features like Elasticsearch and other capabilities, including onboarding and enterprise support, to correctly address scale milestones, to keep the user experience and workflows fast and productive, and to maintain the Mattermost brand.
I’ve personally been on calls watching an administrator show an 8 second lag from keystroke to input because the enterprise scale features weren’t used, and I sometimes feel like that experience alone enough to require commercial support beyond safe limits.
Third, it moves larger deployments to either our commercial or nonprofit offerings earlier in their lifecycle. Ideally a free Mattermost deployment grows to size where it’s value is clear, and the administrator can then purchase a commercial license at a reasonable deployment size, or apply for a nonprofit license if applicable. Without safe limits, some deployments grow to a size where the initial purchase with a very high user count gets far beyond what they’re able to get through procurement. Then the administrator can become stuck without the enterprise, support and escalation capabilities they need to manage operational risk, and Mattermost doesn’t have the commercial relationship needed to ensure the deployment’s success.
Fourth, this change increases investment in Mattermost for mission critical work. When Mattermost champions in critical infrastructure enterprises are enabled to make the business case for the commercial version at the right time, at a reasonable deployment size, the revenues from those purchases accelerate our ability to invest in building great software and accelerating the success of our champions and end users.
Most importantly, we can increase trust from critical infrastructure enterprises that we’re focused on their success. Having clarity that large deployments of Mattermost are intended for mission critical work on our commercial version at scale (whether that’s DevSecOps, security operations, Command and Control or other use cases in vital organization), protecting our brand from issues impacting free deployments that have gone far beyond safe limits, and having more revenue to invest in our platform are all additive to accelerating the success of mission critical work in the world.
The trade-offs in this direction include a) potentially losing some large deployments of our free version, b) potential brand impact from segments of open source users, customers, partners, internal staff and contributors that don’t agree with the above, c) potential trolling in different forms, rather than constructive dialog and discussion.
We’re still working on the right levels of safety warnings versus limits, and what’s important is that we also work with the largest free deployments to ensure a smooth transition.
There isn’t yet an announcement on these changes, specific limits, or release dates, and we hope to have one as the decisions and designs are finalized.
The major version change to signal a significant change impacting our community has been considered, but given safety limits don’t impact ~99% of free deployments, and that 50 users is the recommended size for free deployments, it could cause more confusion than clarity.
Based on anonymous telemetry fewer than 0.8% of unknown deployments are more than 500 users, and higher numbers, like 10,000, would be closer to 0.1%. This might be a little biased as larger deployments are often within self-sovereign private and air-gapped networks where anonymous telemetry can’t function.
To answer your questions in the context of the other thread:
@bbodenmiller: it appears they actually haven’t been implemented yet?
Correct. Since December we’ve been asking for feedback through this user forum, which gets over 300,000 page views a month, and as the user limit functionality is finalized we’ll have an announcement probably shipping warnings at the 10,000 user level in February and functional limits potentially in March, and then looking at lowering the limits in future releases. The PRs you mentioned are some of the early work in progress.
@bbodenmiller: Also regarding “moved to implementing user limits after speaking to our largest deployments … we received surprisingly positive feedback”, where was this feedback collected? There are only a few public mentions all recently that I can find of this proposal and any private interactions are likely largely with existing paying users who would be unaffected and frankly probably benefit from the change despite it going against generally agreed upon on source practices.
You’re correct. The large deployments were supportive because of the benefits from Mattermost accelerating its ability to invest in the platform and invest in their success with a larger commercial customer base.
The intention of safe limits is to have free Mattermost deployments adopt automation and scale capabilities earlier in their lifecycles, with Mattermost’s customer-facing teams helping them make the business case to stakeholders and to smoothly transition.
Until these changes are made, we’ll continue to have situations where Mattermost champions are in “dilemmas of success”–deployments that have grown the free version far past safe limits, and having higher seat counts, high adoption, and accelerated workflows paradoxically increase the complexity and time needed for procurement.
@bbodenmiller: Regarding, “It moves larger deployments to either our commercial or nonprofit offerings earlier in their lifecycle” the limits may make sense for new instances and with various warnings along the way so they are not one day caught by surprise when the instance can no longer accept new users. For existing instances such limits just appearing one day will be extremely frustrating and fall on to the shoulders of poor system admins with no purchasing influence.
100% agree.
We’ll probably start at a crazy high number like 10,000 users. There’s no admin at that scale that isn’t reading the changelog ahead of an upgrade.
As the change is finalized and rolled out, it’s imperative to Mattermost that we support larger deployments so the user limit change doesn’t disrupt operations.
For deployments that aren’t running their own forks, we want to be ready for Mattermost’s customer-facing teams to provide bridge solutions for free deployments running beyond safe limits, and help champions make the business case to stakeholders to join our commercial community, and ensure the platform performs resiliently at scale, as designed.
Our customer teams have helped Mattermost champions at the world’s most vital organizations win the support of their leadership on Mattermost for self-sovereign collaboration, DevSecOps, out-of-band communications, and Command and Control among other mission critical use cases.
No one wants large free deployments to be degraded or discontinued. At the same time, we are prioritizing our brand, our end users, and accelerating the world’s mission critical work by building a larger commercial customer base across critical infrastructure enterprises.
That said, we can only speak to the deployments who choose to engage with us, through community or commercial channels.
To broaden the feedback, we’re inviting commentary here, on our public forums, which have over 300,000 page views a month. We’re looking for input and dialog from deployments we may or may not know exist.
@bbodenmiller: It’s probably worth noting that in many cases it may drive them to competition like Slack and Microsoft Teams instead though.
By joining Mattermost’s commercial community, deployments will have access to Mattermost integration for Microsoft Teams, along with AI acceleration from the Azure AI platform.
In October 2023, we announced our collaboration with Microsoft’s Defense solutions group to enhance Mattermost’s interoperability with the Microsoft in mission critical environments.
Microsoft: Another example is a new collaboration with Mattermost and MobiusLogic. Together, we are working to deliver improved collaboration and workflow solutions for national security customers built on Azure’s secure cloud and leveraging Azure AI capabilities. The solution stack is built on the Mattermost OpenOps AI experience platform connected across Microsoft’s core platform services including Azure AI Services, Microsoft Entra ID, and Microsoft Teams. The hub offers a highly interoperable collaboration experience with advanced levels of customization, integration, AI-acceleration and deployment across private cloud, public cloud and air-gapped networks.
Mattermost is often deployed as a supplement to Microsoft Teams.
By connecting Microsoft Teams to Mattermost, or running Mattermost along side, central IT solves a host of problems, from needing a self-sovereign platform to address regulations and data control complexity (e.g. including addressing Schrems II in Europe), to out-of-band communication for security organizations that might need to run incident response and mitigations concerning the M365 platform, to DevSecOps organizations that need to focus on technical workflows and integrating to rapidly and dynamically changing toolchains and in-house systems.
Again, @bbodenmiller, really appreciate your input and feedback. Thanks again for asking publicly so we can share answers with our whole community.
Very open to other feedback, per the title of the thread,
In my opinion this is way too fast of an escalation for such an impactful breaking change. Many large instances will be on a 3 or 6 month upgrade cycle and thus will not even notice the in-product warnings and have a chance to give feedback before the limit is implemented. Will the blog get a post asking for feedback? In conversations with folks I’ve heard of more people watching the blog than the forum. Would also be good to add warnings of incoming change to Pricing | Mattermost which in multiple places states unlimited for free version.
Furthermore, must organizations budget on an annual basis so it’s unlikely they could make a purchase in just a few months.
For what it’s worth, I’ve seen the open source version run much better than this when running on appropriately sized hardware.
Would love to see above addressed. I haven’t seen precedents for such limits in open source software. I know GitLab for example is against such limits, only locking entire features behind the paywall.
I believe most large instances of Mattermost are either running to avoid the need to buy Slack or Microsoft Teams, or were setup before those SaaS solutions were viable options for various reasons (e.g. data sovereignty requirements). In the case of Mattermost implementing limits, I suspect many will succumb to central ITs desire to move to either Slack or Microsoft Teams (both available at lower price points) leaving Mattermost for only extremely niche use cases which would likely be under the proposed user limits & thus still may not result in a sale and thus the limits not achieving the stated goals.
A better option may be to allow different tiers per user. Imagine an instance with 10,000 users of which 1,000 have the business justification for paid features and the other 9,000 do not. Buying 1,000 seats is a lot easier purchase than 10,000. Over time that could increase from 1,000 out of 10,000 to 2,000 out of 10,000 and so on which is a lot easier than going from 1,000 to 10,000 overnight. I understand some features are at an instance level like HA so the unpaid users would get those for free. Many instance level features could still be flagged behind the paywall (e.g. only paid users get elasticsearch search experience).
For the sake of achieving your goals but not making it painful for those of us who maintain an open source fork: Please consider adding a build-time setting/flag for this so one can set an env var during build to change the hard-coded 10000 limit to something higher or simple have a boolean flag to disable it.
That way you have the 10000 users limit default in your official builds, and maybe even in default Open Source builds (assuming you want this to be enabled by default by distro packages who build from source, since that may be a valid source of future customers after all). But by adding a build flag to allow the maintainer to consciously - yet still easily and without the annoyance of git merge/rebase conflicts - opt-out from the user limit you make their life easier. And I think this is also beneficial for you because some of those people who run such large open-source instances may actually give you valuable feedback!
(PS: I think you know who I am and which instance I run @it33_mattermost considering we briefly met mid-november - just to indicate that the ahove comment is actually from someone who maintains a large instance and not from some random person in the interwebs ;)).
Thanks @ThiefMaster, certainly appreciate your partnership, and your feedback, and getting a tour of your facilities last year. It was quite incredible to see the impact Mattermost has had in your world, and it was super fun dropping by while I was in town.
There’s no intention to make opt-ing out of safe limits unnecessarily complex for forks.
For Feb we’re shipping a warning message for safe limits at 10,000 users to the administrator, and we can look into shipping something configurable in the build process when we implement the actual limits in March.
What’s important to us is that organizations running Mattermost at scale without our commercial or non-profit builds are doing it responsibly, and responsibly managing an open source Mattermost fork is a clear signal of capability.
For others joining the thread here, I’ll just quote from above for context on the 10,000 user safety limit as it applies to forks:
I empathize with any Mattermost champion impacted by Team Edition safety limits at 10,000 users and feeling “betrayed”. I really feel bad about this, because it’s unfair.
While Team Edition is recommended for up to 50 users, we didn’t have safety limits previously, so adding safety limits at 10,000 users, even though that number is extraordinarily high, it does impact a handful of deployments who have actually done an enormous service for the Mattermost community by driving adoption to such gargantuan levels.
We owe you an explanation. We owe you a path forward. I’m trying to be thorough in this forum response to provide both, with context.
My hope is when this is in the rear view mirror that it might be an opportunity for a different kind of success in the careers of our champions. The challenge we see with very large open source deployments is they can wobble after the founding admin moves on. Being the single point of failure of a massive, mission critical system is an enormous and pressurizing responsibility for many.
My wish is that we can support this small number of elite and heroic champions and flow towards a shared vision of enduring success and buy-in for what they’ve achieved.
Part of this is building a relationship with Mattermost’s customer organization and progressing on a healthy path towards the enterprise-level infrastructure, support and investment to enable Mattermost deployments endure and flourish in the long term.
It sounds like the question above is asking about reasons, principles and impact, so I’ll try to cover that with fidelity and with examples, and forward pathing, while also providing context for the broader readership:
What is Mattermost Team Edition?
“Mattermost Team Edition” is a Linux binary compiled by our commercial organization from our open source code base and provided under an MIT license.
It’s named “Team Edition” because it was intended for a single team. In a team, everyone knows everyone. They trust each other. They work together collaboratively. Team Edition has a recommended size of 25-50, because beyond that size, we’re not really talking about a single team.
When an administrator asks for Microsoft Entra ID integration for the open source Mattermost Team Edition so users who don’t know each other can be identified from attributes automatically pulled into the system, it’s clear that it’s not a single team use case, and it should be an enterprise capability in our commercial version.
That’s the simplicity that comes from “Team Edition” naming.
Even with a max recommended size of 50, it’s probably fine going to a couple of hundred users without material issues, but 10,000 is kind of bananas. We are putting in “bananas-level” safety limits.
Why does Mattermost Team Edition exist?
The purpose of Mattermost Team Edition is to drive brand awareness of Mattermost in critical infrastructure enterprises by enabling hands-on deployment and usage.
We provide it under MIT license, which a lot of developers are approved to use. It works with PostgreSQL, which is a common and standard open source database, enabling the administrator to control 100% of the data produced on the system.
Still, it’s very hard to get into many critical infrastructure enterprises, because a lot of them actually ban open source by default.
For those organizations to use Mattermost Team Edition, there’s often a series of security reviews and analysis on our open source code base, including automated scanning, and sometimes multiple groups in parallel need to independently sign-off.
These reviews typically happen without our knowledge or interaction. People are reviewing our source code, they’re going over our website, they’re researching me, our engineering and security leadership, the public information shared by our contributors, and they’re assessing the current and future risk of bringing our open source code into mission critical environments.
We invest a lot in getting Team Edition on the “approved” list at many of these organizations, and in building the brand of Mattermost in trust and safety.
In the context of all we’re investing in brand, introducing safety limits at 10,000 users reduces brand risk from user experience and resilience issues on the largest free deployments of Team Edition that are running without the requisite automation and scale capabilities, and without commercial support.
How do safety limits clarify and accelerate Mattermost’s open source work?
As a commercial open source company our intention is to accelerate investment in our platform and open source work over time, by growing our commercial customer base and expanding our offerings.
Part of this work is clarifying the full benefit of our commercial subscriptions, offerings and operations relative to our open source work. There’s three pieces here:
For anyone who’s contributed to our open source repositories, I’d welcome input and feedback on this thread, this section, or any section.
In my mind, open source software is available for anyone to copy, modify, compile and use.
The value exchange in open source communities is typically to have open source contributions be available in the open source code base, and having a process by which contributions are publicly acknowledged and traceable through the contribution process, which we do via GitHub.
I think of different limits set as constants or variables as common in software, including open source software. Adding the capability to define a maximum safe limit for user count in the code base doesn’t seem controversial to me, but the point of this thread is to hear feedback, and uncover blind spots so input is very welcome here.
I understand that if we compile a specific version of Mattermost’s open source codebase with a safety limit set at a number some members of the community don’t agree with–in this case 10,000 users for a compiled version that recommends 50–it will likely be frowned on by anyone impacted, because it can make their work more difficult in the near term.
At the same time, I don’t know if adding a setting in a specific compiled version of an open source code base would be seen as an infraction on open source principles, since you anyone can compile their own version with any settings they choose.
2 - Mattermost Team Edition Linux binary
Team Edition is a compiled from our open source software by our commercial team, and provided with an MIT license and an array of trust and safety and initiatives, for the purpose of expanding Mattermost brand awareness in critical infrastructure enterprises through hands-on usage.
Our priority is accelerating the world’s mission critical work, and Team Edition is our brand ambassador for achieving that purpose.
The Team Edition experience is optimized for a single team, around 25-50 users. While it’s probably okay to scale to maybe a couple hundred or even a few hundred users on Team Edition, our intention is to have Mattermost champions start considering Mattermost’s commercial offerings for automation, scale and enterprise support capabilities around the time deployments grow past the recommended size.
When hundreds of users in a critical infrastructure enterprise have adopted the free version of Mattermost for workflow, it’s typically straight forward for Mattermost champions to collaborate with our customer-facing teams in developing a business case for IT, finance, procurement audiences and flowing through the organization’s infrastructure planning and budgeting processes.
There’s a host of spend categories where Mattermost excels and a straight forward mapping to budget categories in critical infrastructure enterprises, including security operations/out-of-band communications, engineering/DevSecOps infrastructure, compliance/self-sovereign collaboration, mission systems/Command and Control, digital workplace/Microsoft Teams enhancement, among other options.
If Mattermost is fulfilling its purpose, and accelerating mission critical work in critical infrastructure enterprises, it’s not uncommon to find department-level budget for a few hundred users after making a clear business case.
In addition to scale and automation, our commercial offerings provide enterprise support, federation, administration and compliance, resilient Kubernetes-based operations with high availability and fault tolerance deployment, and a host of capabilities for serving tens of thousands of users over time.
Most importantly, the commercial offerings provide a relationship between Mattermost and it’s largest scale deployments, which have the greatest impact on our brand.
When a Team Edition deployment grows far past its intended size and without the capabilities designed for that level of scale, we have seen some rocky end user experiences, including extraordinarily long page load times that make systems nearly unusable, extended lags between keyboard input and display, freezing of displays, in addition to foundational deployment issues.
These examples of degradation often happen when the original Mattermost champion has moved to a different role or been promoted (most Mattermost admins are quite strong, and high performers), and a new owner needs to manage the system without training, support and without the requisite product capabilities.
Without an experienced admin, and without commercial support and without enterprise capabilities there’s very high risk of Team Edition either being a zombie instance with degraded user experience, or being shutdown completely.
With a commercial relationship the legacy, value delivery and brand of Mattermost deployments endures and expands. Mattermost customer success teams can support transitions to new admins, guiding them with training on the system, understanding system capabilities, continually improving operations and value and making the business case for continued and extended support from IT, finance and procurement stakeholders.
Part of implementing safety limits in Team Edition is to draw attention from all Mattermost champions running free deployments to understand that there’s a healthy path to having their Mattermost investments endure long past their tenure, to make scale, administration and automation dramatically easier, to influence a meaningful, impactful commercial investment from their organization, and to accelerate the resilience and success of their enterprise.
That’s kind of why this forum response is so long and detailed.
Adding safely limits at 10,000 users to the compiled Linux binary we call Team Edition doesn’t really impact anyone who compiles from our open source code, since it’s straight forward to change (see top of this post).
It does impact Team Edition deployments over 10,000 users, and Mattermost’s customer-facing teams will be fully supporting those deployments to avoid any operational impact in the journey towards commercial capabilities.
A key objective of this change is to enable Mattermost champions who have invested so much into their deployments to have their achievements endure, expand and leave a legacy of success in their organizations and in their careers.
3 - Mattermost Enterprise Edition
The third component to discuss here is the commercial version of Mattermost, which provides the commercial relationship, support, account management and broad array of enterprise product capabilities to not only scale but to flourish.
If you’re a Mattermost champion in a critical infrastructure enterprise who’s got a few hundred users on Team Edition, and you’re seeing the system used for mission critical work, and thinking about the future scalability and legacy of the system, it might be the right time to consider starting on the path to joining our commercial community.
The section above includes guidance on how to contact Mattermost’s customer-facing teams as well.
Ideas and input are very welcome. I need to be a little careful in replying here as our customer-facing teams are really the ones that work through these kind of discussions.
Right now the safety limit is at 10,000, so for any deployment where that limit is an issue, I think the first step would be to understand the value of Mattermost for all the current users, and which organizations would budget to enable those users, if any.
Once the data is in, it might be easier to consider different options.
This is an important issue for our largest free deployments, which is why we’ve opened the discussion several months before the changes are planned to be shipped, and have asked for feedback on the change so we can address the top issues when we finalize and announce.
@bbodenmiller, I hit the character limit on the forum for a reply, so trying to answer your other questions here. I really appreciate your feedback, which was requested on this thread, and I’ll be trying to cover down on your points.
This thread is getting long so I might intentionally repeat/adapt some of the context from our previous discussions to err on the side of clarity.
There’s some risk of me getting called out for sounding like I’m repeating what I said before, but repeating some statements in a long thread is how I’m hoping to de-risk people missing part of what I’m trying to share.
The intention of safe limits is to have free Mattermost deployments adopt automation and scale capabilities earlier in their lifecycles, with Mattermost’s customer-facing teams supporting champions in making the business case to their stakeholders and to smoothly transition.
Until these changes are made, we’ll continue to have situations where Mattermost champions are in “dilemmas of success”–deployments that have grown the free version far past safe limits, and having higher seat counts, high adoption, and accelerated workflows paradoxically increase the complexity and time needed for procurement.
We thought of 10,000 users as an “insanely high” number for the starting point of safe limits, for a free product who’s recommended size is 50. The intention was to have impact limited to a very, very small number of deployments on the planet, which we could work with directly to ensure they’re continuing to smoothly operate.
In my mind, in running 10,000 users without enabling automation and scale capabilities, the Mattermost brand has the most extreme risk exposure: a) more things can go wrong, b) more end users impacted, c) longer time to mitigate the issues with procurement and implementation changes, d) geometrically greater brand risk given a * b * c.
So at the 10,000 user starting point, our thinking is we have a geometrically smaller number of deployments impacted (we estimate ~0.8% of free servers have more than 500 users, and 10,000 is nearly zero), and a geometrically greater risk.
That said, I’d like to ask @bbodenmiller a clarifying question–with the caveat that this isn’t something we’re necessarily committing to doing, as there are many stakeholders involved in this–but for you or anyone in the community:
If we shipped user limit warnings at 10,000 in Feb, for admin only, and functional user limits were at 25,000 for March, would anyone consider that a breaking change?
Whether it’s 10,000 or 25,000 users, if the upgrade cycle is 3-6 months, and the functional limits to the free Team Edition ship in March, then it sounds like it’s around 5-8 months from now until when users above 10,000-25,000 need to be deactivated from the system, or a bridge solution to a proof-of-value environment can be deployed, working with the Mattermost customer team.
Did I get that right?
I can’t personally conceive of running at 25,000 users on Team Edition, without scale and automation features enabled, without an incredible number of things breaking and being destructive to the brand.
I’m trying to understand if the issue is user limits in themselves, or certain free instances being impacted.
Regardless of the answer, it’s imperative that our customer-facing teams are connecting with our largest free deployments–especially ones we don’t know about yet–so upcoming changes don’t impact operations for any instance.
Side note: I think there’s going to be a safe limit user limit setting so high that effectively no one is impacted, and hence it’s not a breaking change. The analogy here is SpaceX Starship. My understanding is when getting approval to fly an experimental 100 ton rocket over the ocean, there was a concern that the falling debris from a failure might hit a whale. When the math was done, the surface area of whales relative to the surface area of the ocean was effectively zero. So it’s kind of the same here, there’s some super high limit we can use as a starting point that won’t hit any actual instance, and therefore wouldn’t be a breaking change.
It’ll discuss our focus on critical infrastructure organizations with our enterprise offerings, also discuss our nonprofit self-managed license for communities below 1000 users. We’ll also be potentially announcing a new cloud offering for nonprofits over 1000 users, very steeply discounted, with a value exchange where they may receive earlier versions of new improvements to share their feedback ahead of rolling out to broader audiences.
We’ll be discussing the changes in our free offering as well, with recommendations on securing deployment configurations across private networks and behind VPNs to provide a layered security foundation for the platform. We’ll also be sharing about safe limits rolling out, starting at 10,000.
There’s not a mechanism on the blog to provide feedback on the website, but we can look into linking to the Mattermost forums, or this thread specifically.
Interestingly the reason we had “unlimited” on the pricing page was that we used to have a cloud freemium version limited to 10 free users.
We iterated on the 10 user limit in free, and put “unlimited” for users in, but limits in other areas.
Then the cloud freemium version was discontinued, so there’s only the commercial cloud offerings now, but I think the pricing page needs to clean up the artifacts, because all versions–including paid versions–should have safe limits depending on their configuration.
Appreciate the feedback here, we should remove the out-of-date mentions.
100% agree.
I’m personally committed to ensure we successfully bridge anyone impacted by this change.
It’ll be unsurprising to our customer-facing teams that transitions can take more than a few months.
Sometimes it feels like the larger the purchase, the longer it takes, even when you’ve got a ton of end users using the system, and more users at the doors asking for access.
Often it’s engineers, technical teams and operators driving Mattermost adoption. These are the folks at the core of every critical infrastructure company, and we’re focused on accelerating their most vital workflows.
When Mattermost transitions from free to commercial early, it’s an easier path because it’s within discretionary budgets of technical leadership, there’s validation from tech teams that their workflows are faster, that their transparency and focus is higher, and error rates are reduced–because Mattermost is built for for operators and technologists.
Supporting Mattermost champions in making the business case, and winning the support of stakeholders is what our customer-facing function is built to do.
This includes all the support to keep things operational including figuring out bridge solutions, proper sizing, architecture selection, roll out planning, trial licenses, migration acceleration, and so forth.
In enterprises, when a free Mattermost deployment hits 100-500 users, IT often declares it a “successful pilot”. Then either the champion or procurement contacts Mattermost, or one of our resellers, and works with a mix of our online documentation, our system integrator community, and Mattermost support and technical account management teams on upgrading to one of our commercial versions, accelerating automation and scale, and to smoothly roll out to internal users.
There are larger transitions in the 1000s that have been a little more work. For a 10,000 user transition from free to commercial it might be a mix of pilot-to-production and upgrade-for-scale flows. I think the best outcome would be landing on the Kubernetes operator in a scaled, high availability environment so all the future management would flow smoothly.
That’s very good to hear. We’ve seen some really painful ones.
At the same time, no one really calls us when everything is working well, so our data isn’t super normalized.
One of the reasons of setting safely limits at 10,000 users on the Team Edition was that we didn’t expect IT departments to start considering a massive Microsoft Teams or Slack purchase, or exiting Mattermost all together, when the limit is astronomically high.
A few thoughts here:
Developing a free-to-paid business case at 10,000 users.
If I was a CIO and someone told me there was a piece of open source infrastructure that had a new limit of 10,000 users, coming down over time, I imagine I’d be like “Okay. Sounds like we get 10,000 free users. Figure out who to remove so we can stay under the limit, and let me know when they announce limits coming down, and we can deal with it then.”
What we intend for safety limits to trigger, both at 10,000 users and in future at lower levels, is an assessment of Mattermost’s value among its user base.
This happens regularly in our paid deployments for initial purchase, for license expansion and for license renewals. There’s a process to speak to all the organizations using Mattermost to understand:
a) why they want it,
b) what’s the benefit for them of continuing to have it, and
c) whether the organization’s would use their own budget to buy seats.
Some orgs may or may not have their own IT budgets, so sometimes it’s a “what if” assessment.
Through this process, the business value of Mattermost seats is established and the data can be provided to Central IT for review and budgeting.
In some enterprises, Central IT works through a charge-back system to group buy licenses for all the organizations that want them, and then get reimbursed by organizations using the shared services. In other enterprises, Central IT might just allocate the purchase from a shared services budget, or a departmental budget (e.g. product and engineering), or there could be a mix.
Meanwhile, it’s a priority for Mattermost’s customer-facing teams to support Mattermost champions in working with their organizations through this process, ensuring there’s no operational disruption, as well as sharing patterns, practices and materials to speed success in the commercial journey.
Why buy Mattermost, when Microsoft Teams and Slack are available?
I think Microsoft Teams has an estimated 300 million users right now, so I can certainly see the perspective that everything else is “niche”, perhaps Slack included.
What I can share publicly is what’s available on our website, which is the use of large Mattermost deployments for mission critical work.
This includes the U.S. Department of Defense using Mattermost for operational ChatOps in complex and real world environments, and the use of Mattermost in one of its largest-ever readiness exercises for providing airlift, aerial refueling, aeromedical evacuation, and humanitarian and disaster assistance.
We also enable critical infrastructure enterprises internationally, including utilities like RTE who manages France’s national power grid.
So my perspective is slightly different.
I think of Mattermost’s focus on accelerating the world’s mission critical work as an important and large market, and a meaningful mission for our company and our community.
When the work of an organization is vital to the safety of a society, there’s an elevated need for resilience, and for platforms dedicated to the success of operators and their technical teams, which can adapt to their specific needs.
This is the business case made over and over again within our commercial customer community.
Resilience in Microsoft Teams environments
In the context of resiliency, Mattermost’s ability to supplement Microsoft Teams is an essential service for many customers who run both.
My co-founder and I are former Microsoft engineers, that worked out of Redmond, and we have enormous respect for many of the security professionals that work there. At the same time, the cyber pressure Microsoft faces is extraordinary and increasing, due to the growing magnitude of confidential data they are protecting, advancements in offensive cyber capabilities and in the sheer ferocity in the newest generations of attacks.
It’s difficult for many critical infrastructure enterprises to get comfortable storing 100% of their confidential communications in Microsoft Teams as a multi-tenant shared service. Especially when it’s operating on a shared code base on public cloud with known points of ingress and egress, and uses availability keys in its encryption protocol.
I’ve talked to some security professionals who share that for safety reasons they need to operate under the assumption that Microsoft Teams is breached at all times, and any incident response or discussion of confidential vulnerabilities data needs to happen in an “out-of-band” system.
Part of this is the principle that security operations should be separated from the environments they’re protecting. So if security professionals are protecting a network, they should be on a separate network. If they’re protecting Microsoft Teams, defending against threats and responding to breaches or new vulnerabilities potentially in M365, they should be on a platform out-of-band from Microsoft Teams.
The other part of this is the principle that the effort that goes into a breach is proportional the economic value of the breach. At 300 million users, the economic value of breaching Microsoft Teams is astronomically high for attackers, compared to the value of breaching a self-sovereign Mattermost system used by only a single target.
Also this doesn’t apply just to Microsoft Teams, if you have any primary platform for collaboration that your security operations is tasked to defend they should have an out-of-band platform for security operations.
As a counter example, when you read through the public SolarWinds disclosures, it seems clear the security organization didn’t use out-of-band communication, so when their primary collaboration system was breached without their knowledge, information about vulnerabilities in their systems and in their customer base, as well as their response plans were flowing to attackers, compounding the problem and creating a cycle where recovery was somewhat impossible.
Beyond the security perspective, for resiliency in mission critical operations, out-of-band scenarios also apply to platform engineering, SRE, DevOps, and DevSecOps scenarios when organizations want control of their own uptime SLAs.
From this context, for any critical infrastructure enterprise running Microsoft Teams or any other primary collaboration platform at scale, there is an opportunity for Mattermost to provide out-of-band support for resilience in addition to workflow acceleration.
Focus and Adaptability to mission critical workflows
Mattermost provides operators, technologists and developers an environment focused on accelerating their vital workflows, increasing their visibility, and reducing error rates and blind spots in the delivery of mission critical work.
For technologists and DevSecOps practioners, Mattermost’s ability to integrate with systems like GitLab, to be customized, to integrate with custom tooling, in-house systems, custom security and compliance requirements, as well as connecting into DevSecOps infrastructure inside and outside the Microsoft portfolio all contribute to providing the focus needed to accelerate technical and operational success.
For operators in mission critical and high compliance environments, Mattermost’s open source code base and self-sovereign design lets organizations adapt Mattermost to nearly any purpose–deployment in private and air gapped networks, integration with custom compliance tools, customized federated communications across security boundaries, deep AI automation and a host of scenarios and capabilities that centralized SaaS platforms would struggle to provide.
In contrast, general collaboration tools like Microsoft Teams are designed to serve “everyone”, and there can be a level of information overload for end users, a level of request overload to Central IT for supporting specialization, and a level of system limitation that makes mission critical work significantly more difficult to complete.
Without the right tool for the job, without the right support for the vital work of operational and technical organizations, the risk of blind spots, errors and delivery delays can elevate. This can lead to risks in retaining the strongest engineers, and recruiting new stars. Unaddressed, these risks all together can snowball towards compromising operational resiliency and success for the enterprise itself.
Mattermost is built to address these risks, both independently and as a supplement to Microsoft Teams through our interoperability investments: Mattermost and Microsoft Solutions
Circling back to your question on Slack, we don’t see them that often in critical infrastructure enterprises, and my understanding is our value propositions are quite difference.
Mattermost take a different approach, with a focus on critical infrastructure enterprises and accelerating the success of operators, technologists and developers in their mission critical work, and offering self-sovereign deployment and full control of private data and IP.
@bbodenmiller thanks again for the feedback, which is what this thread is for. Very open to more input from you or any reader.
Thanks for the detailed response @it33_mattermost - it’s a lot to consume.
IMO it’s still a breaking change even if it affects minimal instances.
Is the limit based on monthly active users, concurrent users or “activated” users? Long running instances could have a lot of “activated” users with much much less monthly active or concurrent users.
Easier said than done without robust user management features not present in Team Edition. With the limit presumably based on activated users rather than monthly active users, this means once a user is deactivated they cannot self-serve return without admin intervention creating a host of problems. Sure it may drive license sales but it may just drive people away.
Maybe but with the implementation timeline discussed here (limit starting in March release) it seems like the limit will be in place before some admins are even seeing the warning let alone get ahold of Mattermost, Inc. to discuss options. It is not uncommon for admins to skip releases.
Environmental Law is outside my area of knowledge or expertise
@it33_mattermost this is an interesting explanation of Team Edition that I have not seen (or simply missed) before. Pricing | Mattermost hints at it, though as previously discussed “Unlimited Teams” confuses the point. Mattermost editions and plans — Mattermost documentation doesn’t really seem to communicate the point that it’s designed for a single team until you get to plans which at skim I would have assumed would only apply to Enterprise Edition.
I agree the name Team Edition given the explanation makes sense but I’ve always just seen it as Mattermost open source and Mattermost paid edition - I’ve only used it via GitLab Omnibus package which doesn’t really seem to call out the intended use case or edition - GitLab Mattermost | GitLab. Interestingly, upon looking now I see the download for Team Edition is even buried away quite a bit and not mentioned much. Also unclear how or if it can be used on Kubernetes.
If Team Edition is made for a single team it seems it may have made sense to restrict the multiple-Teams capability to Enterprise Edition when introduced. I would highly recommend not making that change now. This thought exercise has helped clarify what bothers me most about the 10,000 user limit: it’s moving/restricting a capability currently in the open source version to the enterprise version. Removing/restricting capabilities in open source tier in favor of paid tier is something that a number of companies backing open source products have vowed not to do, like GitLab (stewardship promises):
When a feature is open source we won’t move that feature to a paid tier.
I had hoped Mattermost, Inc. operated under a similar promise given how similarly the two companies operate.
Now if we’re talking implementing an overridable safety limit that could be a bit different conversation (same story if it were just an annoying safety warning to admins). GitLab promises “the open source codebase will not contain any artificial limits” but they do include sane default limits which can be overrode (though not always easily) if admin believes/sets up infrastructure to handle it. If Mattermost, Inc. were to allow the safety limit to be changed or override (without re-compiling the code base), I don’t think this is such a controversial change (whether it’s breaking or not is still up for debate). Without such override I see the change as ultimately commercially motivated, and not in line with the promises other companies have made in regards to the open source software they maintain.
Why does Mattermost, Inc. compile user limits into its Team Edition offering?
The purpose of Mattermost Team Edition is accelerate brand awareness of Mattermost as a collaboration platform for mission critical work by providing hands-on experience with our offerings.
For Team Edition, a “team” is a group of people that know each other, trust each other, and who work well together. Size is intended to be 25-50 people at the top end. It’s okay to go a little higher, but generally we’re creating Team Edition for teams, not enterprise-wide deployments.
Since November 2023 we’ve been publicly sharing plans for warning messages when Team Edition was significantly exceeding these levels, and sharing our intention to add user limits, which will lower over time. This included announcing intentions to ship a warning at 10,000 users Feb 2024, and a functional limit in March 2024.
These changes are rolling out now, with a ERROR_SAFETY_LIMITS_EXCEEDED warning message when users exceeded 10,000. There’s also a functional limit at 11,000, per our announcement.
If you’re compiling Mattermost from its open source repository, these settings can, of course, be changed. The limits only apply to the Team Edition version compiled by Mattermost, Inc. If you’re a non-profit that wants to use version of Mattermost compiled by Mattermost, Inc., please consider our non-profit program.
Our intention is to have a public and open dialog about these changes, and we greatly appreciate @bbodenmiller and @ThiefMaster’s input in this thread. There’s over 300,000 views a month on this forum, and we welcome input from the community on this topic.
So far, this doesn’t seem to be a broad issue in our community.
First, it’s important to share that this feedback is appreciated and certainly considered. Any Mattermost deployment with over 11,000 users is important to our brand and our community, and we want to have an open dialog.
At the same time, it’s important to emphasize the safety limits can be changed by anyone who compiles Mattermost from its open source code base, per above.
Mattermost is typically deployed in critical infrastructure enterprises–organizations identified by governments as essential to public safety, national security or continued operations. This includes defense, financial services, utilities, communications, and vital manufacturing entities among many others.
For regulatory, compliance or security reasons they’ll choose to self-host Mattermost for collaboration, typically starting with a small open source instance, and then investing in commercial offerings as their user count and needs scale. The commercial offering and relationship provides the automation, compliance, administration, scale capabilities and support assurances needed by enterprises, and its a win-win for everyone.
Enterprises want to know the infrastructure they depend on is well-funded and continuing to invest in the success of the platform. When the user limit comes down to around 500 or 250 or even 50, there’s certainly a case that it’s helping to earn the trust of enterprises with increased monetization and investment into the platform.
At the same time, Mattermost, Inc. considers 11,000 users on a system intended for a team of 25-50 to be a material risk to end users, the enterprise and the Mattermost brand, so we use ERROR_SAFETY_LIMITS_EXCEEDED to express that position.
As with all open source work, anyone who would like to see things behave differently is welcome to change or remove the limits by compiling their own version of Mattermost from its open source repository.
Non-profits that don’t have the resources to compile their own version can apply for our non-profit programs.
Is this an issue for anyone else in the community?
This thread has been going on for a few months now, just curious about additional opinions.
Hi @it33_mattermost, thanks for providing insights about Mattermost’s choices and strategy.
As you said earlier, you don’t get much feedback from the community when things go well, which is unfortunate because Mattermost is a great piece of software.
Let me tell you about our instance.
I’m a member of Picasoft, a French NGO that promotes free software, raises awareness about mass surveillance, educates people about how the Internet works and hosts free software as a way to propose concrete alternatives to proprietary software (and surveillance), and as a way to have a more human-like relationship between users and admins. In short, we have interacted with most of the organizations that find their home on Picateam (our instance), and we are able to deal with their questions and problems - we provide support.
Some statistics about our instance : almost 11k users, 2 millions messages. It runs on appropriate hardware and we have no performance problem.
Picasoft has been entirely run by volunteers since its creation in 2016. In France, we are officially recognized of « general interest ». In order to operate, we need about 1k€ per year to support hardware failures and housing costs. We collect this amount exclusively through donations, whether it is from NGOs, companies, public services or individuals.
In short, our instance became a hub for small organizations seeking privacy, so we host a lot a small teams, mostly private teams. Our popularity grew by word of mouth. This scenario is only conceivable precisely because we have a single, recognizable instance. You said that the original intent of Mattermost Team Edition is to host a single team - but for us, it would be inconceivable to spawn a new Mattermost instance every time an organization hears about us, both in terms of sysadmin work and in terms of delay for users.
We really don’t need the features of other editions, so applying for the Mattermost Nonprofit License is not relevant. As you can guess, we have no way to afford any of the other editions (and still don’t need the features).
I do understand that, as a company doing business and seeking adoption, you don’t want your name being associated with slowness, bugs, and so on. But hard-coding limits won’t solve this, because any organization that reaches 11k users and is happy with the Team Edition won’t just drop it and buy another edition. The only solution for us is to fork and compile Mattermost without these limits, which just means more work for our sysadmin team and does not eliminate the risk of your name being associated with failures. No one wins in this situation.
From our point of view, this change is experienced as a need to “fight” against a once friendly piece of software, just like we have to do every day when working with proprietary, bloated software.
I hope this gives you another use case, and thanks again for providing feedback.
Hi @Chostakovitch thanks for your input and your example. Really appreciate your sharing, as it’s been a couple of weeks and we haven’t heard from anyone else.
It’s excellent to hear that things are running well in your deployment.
That said, we have seen extremely bad deployments where it can take more than 10 seconds to type one character because the system has been so overloaded, particularly from millions and millions of messages in the Mattermost Team Edition database, which is far beyond what was intended for one team.
Message history length is a key issue in performance, and we are explicit in our documentation that the Enterprise Search capability in Mattermost Enterprise should be required after a certain volume of messages, but we have seen repeatedly this not followed and the end user experience can be terrible in these deployments.
What’s worse, those deployments don’t realize the slow response is abnormal for Mattermost.
An alternate solution to functional user limits may be to have messages history limits, which could make the experience in oversized deployments more similar to IRC–but which can be turned off for anyone compiling Mattermost from its open source code base.
We really don’t need the features of other editions, so applying for the Mattermost Nonprofit License is not relevant.
We want to encourage non-profit organizations larger than a team to join our non-profit program, which enables Mattermost, Inc. to have a direct relationship and hear feedback on what’s working and not working in the system, performance issues, usability issues and input on our future roadmaps as well.
The non-profit program is a value exchange between Mattermost, Inc. and the non-profit community seeking the benefits of the commercial version of Mattermost, an opportunity to contribute feedback, and to play a role in Mattermost’s purpose of accelerating mission critical work for our government, financial services and vital enterprise customers.
There will be non-profits that want to support this purpose and contribute as part of our community, and helping us to accelerate brand awareness of Mattermost for mission critical work.
There will be other organizations that may not agree with the changes we’re making to the version Mattermost, Inc. compiles, who may not choose to join our non-profit program, and who may choose to compile our open source code base with limits disabled, or even to switch to a different solution.
The only solution for us is to fork and compile Mattermost without these limits, which just means more work for our sysadmin team and does not eliminate the risk of your name being associated with failures.
As with any brand, we do have a trademark policy for protecting the Mattermost name and logo and as an organization forks the code and we just ask for proper usage, as you would with any trademark.
While I realize it may be work for you and your organization, this is an important change for us in building our brand among the mission critical infrastructure community.
Of course, these changes are a difficult decision as it’ll benefit some groups and create extra work for others.
For me personally, having been on calls watching users being nearly unable to type into a search box because admins ignored our documentation and allowed message history to vastly exceed limits, I would choose to concentrate our community to avoid some of the bad user experiences I’ve seen from under-resourced deployments.
Are there any other community members interested in the user limits or message history limit discussions?
There’s 300,000+ page views a month on our forums, I was hoping for more feedback.
There’s fewer than half a dozen people on this thread.
Hi @Chostakovitch, thanks again for your feedback. As an update the thinking now is to have a starting point of message history limits that match the warnings and functional limits in Team Edition documentation, with a warning message at 2.5 million messages in the database and a functional limit at 3 million which will remove the oldest messages that are over the limit from display and search–though they will still be in the database so upgrade to the non-profit offering or the commercial offering can happen smoothly.
I think this code already exists as there we used to have message history limit in the Free Mattermost SaaS offering which was discontinued last year.
Similar to user limits, we might bring down the message history limit over time so we can more clearly define the purpose of Mattermost Team Edition as a solution for a single team, but applying the message history limit to match our documentation seems reasonable.
Thinking right now is to implement this similar to user limits, where community members compiling their own forks of Mattermost can change build-time variables to set the limits however they would like or effectively remove them.
Given the search box operates client side, how is this even possible?
Sounds like useful safety limit if it were a configurable option.
@it33_mattermost why not just admit that these limits and making them hard to change is a cash grab - an attempt to get large organizations to pay for Mattermost that aren’t currently? I stand by my previous comment that adding artificial limits with no overrides is out of touch with the open source ecosystem.
I was on a listening tour call at a critical infrastructure enterprise where the people using Mattermost shared screen to show the issues they were having, which including typing into a search box and having a wait until the characters typed showed up on the screen, before they could even show how long the search results took to come back.
No other tools were allowed in their environment, so they just had to use the system this way. This was in a critical infrastructure enterprise and end users had to deal with this kind of system lag when Mattermost was misconfigured.
The original admin that deployed Mattermost had left, and the new admin wasn’t aware of the scale guidance in our documentation, and their Mattermost deployment had just grown wild: Scaling for Enterprise - Mattermost documentation
There’s been a lot of changes since then, including upgrading to Elasticsearch, and it’s been transformational.
Hi @ThiefMaster, we had initially drafted a change to add build-time settings but an issue arose.
On the one hand, we want to encourage open source deployments of Mattermost that adopt the reciprocal open source license when self-compiling the code base, which makes the source code of the derivative work accessible by all its users.
On the other hand, by adding a build-time variable, which could result in removing safety limits without clearly creating a derivative work, which means reciprocal licenses wouldn’t be adopted. Because of this, we’ve decided to close the PR that would have added build-time settings. Sorry about this. Realize it’s a little bit more work for fork maintainers.
Just to update everyone on the internal discussions the latest thinking is:
1 - Hold off on message history limits at 3 million, and instead have a better warning when recommended message limits are exceeded (there’s already an admin warning at 2M messages, but it’s buried deep in the admin console and super hard to find).
2 - We’ll update the existing admin-only, non-dismissible ERROR_SAFETY_LIMITS_EXCEEDED banner message appear when message history limits are exceeded to make it more clear this is an issue, and documentation will be updated to direct admins to our scaling guidance. This is important in the case where a new admin is inheriting a Mattermost instance they never installed, so they can understand when it’s gone far past its intended design.
If you’re self-compiling our open source code base using a reciprocal license you’re welcome to use the software however you like, including changing the banner, and we’re very open to your input and feedback and to help make things easier when it’s not in conflict with other priorities.
If you’re an enterprise with hundreds of millions of dollars in budget and over 11,000 employees and don’t want to self-compile Mattermost’s open source code base, and want to only use the version that Mattermost, Inc. compiles, and don’t want to invest in scale, resilience and automation capabilities provided by Mattermost, Inc. in its commercial version, that is 100% okay. What Mattermost, Inc. is saying is that in future upgrades to the version it compiles, there’s going to be a limitation in the ability to add additional employees beyond 11,000.
(lots of edits to fix grammar and meaning; as you may have heard, French people are not good at talking english )
From what I’ve read, this thread won’t change anything about the current direction regarding user limits and now message limits - at least it’s nice to chat with you, but also a bit frustrating. Please be ensured that there is nothing personal with you; I really intend to make you feel our point of view, while trying my best to feel yours.
This time I am writing in my own name, but I think I am not the only one who feels that this is ultimately a commercial move. Maybe things would be simpler if presented that way.
In a 2018 thread, @jasonblais seems perfectly fine talking about teams with 10k+ users. The hardware requirements page just tells us to stress the system with a testing tool if teams reach 2k users.
I am also a bit frustrated of this narrative because, while it makes sense in hindsight, neither the documentation nor the forum made it explicit until now. We launched our instance in 2016 (at the time we were even maintainers of the Docker image - hi @pichouk), and we never thought that we would have this conversation one day.
Perhaps you have indeed seen dramatically slow Mattermost instances since then because they grew on unsuitable hardware. Right.
But in that case, why isn’t a warning banner for administrators enough ? You gave the example where “the original admin had left […] the new admin wasn’t aware of the scaling guidelines”. Well, that is the purpose of warnings: to prevent this kind of problems, make information available and to point to the appropriate documentation (hardware, pricing, whatever). No sysadmin would ever deliberately let their instance become slow and unusable. Don’t you think so ? Generally speaking, in the open source world, programs warn people, but always provide a way to bypass the warning if people know what they’re doing (think rm -rf --no-preserve-root /).
Right now, this Mattermost move feels like you assume that sysadmins are unable to correctly use Mattermost, and that they need to be technically hard-limited before they start taking appropriate action. This is not a good signal from my perspective, as it sounds to me like presumption of incompetence. In my opinion, a healthier presumption is to assume sysadmins are competent, not to punish everybody because some instances are dysfunctional.
I do get that point, but please consider the following:
We do not need Enterprise features, and in general we tend to prefer to run FLOSS code on our machines.
We have 10k+ users, so according to the nonprofit Q&A, we have to undergo a special review, which may clash with our efforts to keep our free will.
We are not guaranteed to be able to spend 250$ every 3 years
We have absolutely no insurance that things will stay as they are, and this thread is a perfect example:
What if Mattermost, Inc. refuses to renew the license after 3 years for some arbitrary reason ?
What if the price goes up and we can’t afford it anymore ?
If you change your mind at any point, we cannot go back as some features are like points of no return (e.g. imagine that we finally start to use LDAP integration, but that the next year our application is rejected for whatever reason; what could we do ?)
etc.
On the other hand, the FLOSS version is guaranteed to run forever, even if we have to stick to a specific version because the license would have changed.
I said earlier that our usage of Mattermost is quite special: indeed we have hundreds of small teams, mainly non-profit organizations, public research teams with no budget, very small companies, students, and so on. We really cannot afford to enroll and then be excluded from the nonprofit program for reasons beyond our control, hurting not just us but all those organizations. So yeah, from my point of view our instance is “critical”, but not because we need high performances, complex workflow integrated with Gitlab, clusters with 99.999% availibility, etc., but because a whole heterogeneous ecosystem depends on us to communicate. Just that.
So far, my guess is that you probably want that companies which…
Use Mattermost Team Edition for free
Make a reasonable amount of money
Are using Mattermost for production-ready, critical work
…simply pay for using Mattermost.
This is perfectly legitimate; I personally reject the idea of making money on the FLOSS community while barely giving anything back (hi, AWS) - for that reason I highly value AGPL. I think you see the Team Edition as a trial for, say, a small development team of a random company. As the adoption grows across multiple teams and as the instance scales, you want the company to consider Mattermost Enterprise, because it can afford it and because you, as any company, you need income. As a side effect, the company will also benefit from support, performance, easy scalability and features that are generally useful to big companies, so it is a win-win situation.
My bet (and regret) is that our particular situation (where neither the nonprofit nor the enterprise version seem to be suitable) seems to have no importance in your decision making, since we are not the actual commercial target, despite being a “champion” and despite not being the only impacted nonprofit NGO (Framateam is another very close example).
But still, I wanted to let you know that for us, a small nonprofit organization with tens of hundreds of users and only a few volunteers who already spend a lot of their spare time trying to provide an privacy-friendly, free, humane alternative to (Slack|Discord|Google Drive|Trello|whatever personal data leecher) for small organizations and privacy-seeking individuals, having to maintain a fork mainly because of commercial reasons (and because of an obscure gray area about whether changing a build-time argument triggers the AGPL) is frustrating.
You are licensed to use compiled versions of the Mattermost platform produced by Mattermost, Inc. under an MIT LICENSE […]
You may be licensed to use source code to create compiled versions not produced by Mattermost, Inc. in one of two ways:
Under the Free Software Foundation’s GNU AGPL v.3.0, subject to the exceptions outlined in this policy; or
It is utterly clear that compiling ourselves Mattermost makes the license reciprocal, as the MIT license is tied to the binaries that Mattermost, Inc. produces.
Also, the AGPL makes it very clear that build-time arguments are part of the software:
The “Corresponding Source” for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities.
Hence, it seems to me that the license argument which was used to close the discussed MR is at least very fragile.
Back to the fork hypothesis. We deeply care about the security of our users. We have a bot that warns us as soon as there is a security release (see below), so we can upgrade ASAP by changing a variable in our Docker Compose file.
While it takes time for us volunteers, it is still manageable. Maintaining a fork is a whole different ballgame, and I’m not sure we can apply critical security patches rapidly, having to go through the whole pull-patch-compile-dockerize-deploy process, making our infrastructure more vulnerable.
As a final thought, I would like to argue a last time that issuing warnings with links to various pages of the documentation is sufficient if the issue you’re trying to tackle is really security and performance. As I’ve said, no sysadmin would ever deliberately let their instance become slow and unusable - unless they do a bad job, which is clearly out of your responsibility and power. If a sysadmin does a bad job for whatever reason, I bet that its instance will go through a whole bunch of other problems that you can’t solve with limits. Undergoing hard limits makes me feel a bit paternalized, but more importantly it hurts organizations like ours, that use our spare time to advocate for privacy and FLOSS.
However, please be ensured that I really do value and appreciate that, as a CEO with tons of stuff to do, you take time and care to interact with the community. While expressing frustration and difficulties is important to me, I really have no animosity towards you. Besides, I regret as much as you the lack of participation to this thread. Unfortunately, most of organizations like ours usually don’t even know that such discussion/feedback request happen there - I really stumbled on this thread by chance.
Thanks in advance for reading and, I hope, answering with frankness.
