Mattermost 3.x is a maturing product and we’re considering a move from monthly releases to bi-monthly releases. We’re posting about it here to get early reactions.
This means instead of releasing monthly (e.g. October 16, November 16, December 16, January 16…) we release bi-monthly (e.g. November 16, January 16…)
We’ll continue our existing policy to ship hot fix releases for priority security issues and priority feature issues.
Here are the pros and cons discussed so far:
Pros:
More improvements - The release process at Mattermost is extensive, including intense manual testing and preparations. By reducing the ratio of release-to-development effort, we can ship more improvements in a year on a bi-monthly schedule than a monthly schedule.
Less work for IT admins - It is asking a lot of IT teams to upgrade Mattermost every month, especially in large organizations. Bi-monthly releases mean less work, and more improvements per release.
Fewer out-of-date deployments - All other things being equal, going to 6 releases per year from 12 reduces the number of Mattermost deployments that aren’t on the latest release.
Easier to accept contributions - Monthly releases means there’s very short windows to accept and merge community contributions. Bi-monthly releases make it easier to work with the community.
Cons:
Waiting longer for improvements and fixes - If we miss a feature in a monthly release, it could be out in 4 weeks. With bi-monthly it would take 8-9 weeks.
Possibly longer to react to changes - Right now, it’s natural to plan monthly based on releases, using the latest feedback and data available. Bi-monthly releases may slow our decision making if we’re not mindful.
Having too much to document in one release - Thanks to our community, we ship a lot of new improvements monthly, and our team is under pressure every month to document everything properly. If we don’t have a continuous documentation process properly set, bi-monthly releases makes documentation even harder.
We’re thinking of trying bi-monthly for at least a few releases and if it seems monthly is better we can always switch back.
UPDATED: Added note on reducing upgrade latency, thanks to @JeffSchering.
Maybe look at doing a long term stable release that only gets fixes for bugs and security fixes, and keeping a monthly release for those who install for the first time and for those who always want the latest-greatest. This would be similar to the Ubuntu LTS method https://wiki.ubuntu.com/LTS and the Firefox ESR method https://www.mozilla.org/en-US/firefox/organizations/faq/.
Also, to help with the decision: What metrics do you have? How many orgs upgrade every release? For those who don’t upgrade every release, how long do they stick with a back-level release? Do you have more new users for each release than existing users who upgrade?
For those who upgrade infrequently, what is the reason? Is upgrading difficult? Do they feel that their user base would be unsettled by monthly changes? Is their usage too low to justify a monthly upgrade hassle?
While I can see where you’re coming from on long term releases, it’s probably a different discussion. Right now we keep things simple and ask everyone to upgrade to the latest release, or security update release, which include security updates.
Good point on upgrade latency. That wasn’t on the list, but we’ll add an update.
All things being equal, going to 6 releases a year from 12 releases would only improve upgrade latency. We have download metrics, but they’re complicated because of Docker, Puppet, Chef, Ansible and other community installers and we’ll probably share that in a separate update after we have time to parse things out.
Regarding talking to the community about upgrade latency, that makes sense too–most active community members are upgrading every release, there are a few that are happy with older versions. We really haven’t done a lot of research into folks that are on older builds (they’re also harder to reach, if they’re not active in our community).
We might want to put out a survey some time on our blog… or maybe offer a survey link in the product’s About Page… (though we’d need to think about whether to support different languages).
Just curious, what’s your personal experience with upgrades? Feedback highly appreciated.
Unfortunately I don’t have any upgrade experience. Nor to I have any install experience. Yet :). I’m in the process of doing that now. From reading the docs though, upgrading looks pretty straight-forward and easy to do. The only thing that I can think of, is that the upgrade doc doesn’t yet mention TLS; the draft TLS docs say that you must set Mattermost to bind to low ports each time you upgrade
If you want feedback on the install experience, I’ll gladly do that, probably in the form of a pull request on the install docs. I don’t have a lot of time available for these kind of projects right now so it’ll take a while, plus I want to use MySQL on Ubuntu 14.04 LTS, and the docs currently only cover PostgreSQL.
While I love seeing monthly (read: as soon as possible) updates featuring new features and bug fixes, I can also appreciate fewer upgrades in need to be scheduled and executed (on the admin side) and less release work on the dev side.
This is not a problem of my deployment specifically though, as I have written a script that does the upgrade for me ( ./upgrade_mattermost.sh 3.4.0 ) and a user base (about 100 users in the same timezone) that is not too picky if the chat is not available during lunch break or similar hours ;).
Personally I feel like Mattermost reached a point at which it does not urgently need new features to be released within one month. However as someone comfortable with english and already served with a full translation (german) I can’t say wether this is how the majority of the user base feels.
Especially for users/deployments that are different from mine (in use case / language / user base ) I don’t feel confident saying a slower release cycle is just as fine, as they might be highly anticipating an upcoming translation, a specific feature that’s already scheduled or a fix for a bug impacting their workflow/interaction.
An idea basing on JeffSchering’s suggestion of a split into an LTS and a regular release (which could be quite a lot of overhead I assume):
Keep the monthly release cycle for bug fixes, improvements and features that are highly requested/easily implemented (possibly through a pull request). For major new features, complex changes etc. make a bi-monthly schedule (using every other release of the monthly cycle) as you lay it out in your OP.
While this does combine some cons of both solutions, it also combines pro’s. It would give room for more work on the bi-monthly release and take work off of the monthly release. At the same time it keeps frequent updates for wanting users, but might not make updating as necessary (/mandatory) for already satisfied users.
As an admin i think upgrading is not that hard. I use the docker images and its just a question of updating the images and restarting. We do use a proxy so i have to keep my own set for docker-compose up to date with the main repo. Not as much hassle as a lot of other tools.
Why not change to bi-monthly but have some sort of daily builds. With the “use at your own risk” kind of thing. People that really want the bleeding edge or just want to see if something is fixed in particular, if it works for them they can go ahead. The daily builds don’t have to be all fancy like a normal bi-monthly release.
If people want bleeding edge, they can still use and test it and give feedback rather then having to find an issue when the bi-monthly comes out. Could be a win win… just thinking out loud here.
@reach3r would you be interested in sharing / open sourcing your upgrade script some time?
Might be something people could use or even add-on to. Maybe even as a Gist?
Just a thought.
You make a good point about translation, as the translation team for a new language does need to wait a few weeks longer–as it he case now that Russian translation is complete.
At the same time, for the translation community instead of having to do updates 12 times a year, they only need to do it 6 times a year.
For bugs, if it’s a critical functionality or significant security vulnerability we’ll still have hot fixes. For minor issues, not sure it’s a good use of time to have people upgrade–or to support multiple versions for that.
I think at some point we’ll change the schedule again based on how the project matures, right now we’re extremely excited about the November release and what we can put out in 2 month cycle of continuous development.
@it33 Yes, I thought about it already but did not quite get around doing it so far.
Where would you suggest be the appropriate place to share it? On the forums or someplace else?
Thanks @reach3r! Just a post on the forums would be an excellent start. I think the more people share, the more others want to share and contribute, it’s just positive overall.
) and a user base (about 100 users in the same timezone) that is not too picky if the chat is not available during lunch break or similar hours ;).