Our Experience Submitting “Community for Mattermost” to the Microsoft App Store
The journey of submitting our Microsoft Teams app, “Community for Mattermost,” has been both instructive and challenging. The app enables users to connect to the Mattermost Community as a tab application from within Microsoft Teams and Outlook in both web and desktop environments. Here, we share our experiences navigating through three rounds of submissions and a pending fourth.
Round 1: February 27, 2025
-
Submission Date: February 27, 2025
-
Microsoft’s Response: February 28, 2025. An impressive one-day turn-around time.
Learnings
-
Integration Plugins: Initially, all integration plugins were enabled in Mattermost Community, leading reviewers to feel obligated to test every integration (e.g., Jira, Github). To streamline the process, we disabled all plugins except our AI plugin.
-
AI Capabilities Labeling: Microsoft mandates that any AI features within a tab app be clearly labeled. Content moderation is also required. We ensured AI-generated responses included a disclaimer, provided an email to report inappropriate content, and instructed our AI to avoid answering inappropriate topics.
-
CSP Frame-Ancestors Header: The app needed to include all domains of Microsoft’s O365 applications in its CSP frame-ancestors HTTP header.
-
App Manifest Testing: Every capability mentioned in the app manifest is rigorously tested. For instance, if the app promises access to a community where questions about Mattermost can be answered, reviewers expect interaction. We clarified that responses may take 24 to 48 hours.
-
Minor Wording Changes: Some minor wording changes were requested in the app manifest and submission.
Round 2: March 5, 2025
-
Submission Date: March 5, 2025
-
Microsoft’s Response: March 6, 2025. Another impressive one-day turn-around time.
Learnings
-
AI Functionality: Our attempt to retain AI functionality was unsuccessful. Even though inappropriate topics were rejected, some appropriate topics lacked adequate content moderation. For example, a candidate rejection letter crafted by our AI did not meet Microsoft’s criteria as it did not include a statement about not rejecting candidates for no reason. Due to the opinionated nature of the expected content moderation, we decided to disable AI capabilities until integration with Azure Content Moderation APIs is possible. This will incur per request costs for Mattermost.
-
New Screenshots: Microsoft requested new screenshots for our submission due to minor inconsistencies.
Demo: March 11, 2025
- Live Demo: We provided Microsoft reviewers with a live demo to better understand the app and its capabilities.
Round 3: March 11, 2025
-
Submission Date: March 11, 2025
-
Microsoft’s Response: March 13, 2025.
Learnings
-
AI Plugin Confusion: Despite disabling the AI plugin, the previous AI content and bot accounts remained, causing confusion. We clarified that no new AI content could be generated, but historical chats needed to be preserved.
-
Community Channel Responses: Reviewers did not receive an immediate answer to a question asked in one of the Community channels. We updated the app manifest to indicate that responses typically take 24 to 48 hours.
Round 4: March 13, 2025
-
Submission Date: March 13, 2025
-
Current Status: Awaiting response as of March 17.
The process has been an invaluable learning experience, and we eagerly await the feedback from Microsoft for our latest submission. We’ll provide updates as our submission progresses.
Congrats on the approval, Doug!