Protonmail supports SMTP and mattermost does so too, so after you created an SMTP account and an SMTP token in Protonmail, you can just enter this information into the SMTP server settings in Mattermost.
Did you try that already and if so, what problems did you encounter?
I have a business account with Proton that allows me to generate SMTP tokens to use with third party apps. I still receive the “Connection unsuccessful: Connection unsuccessful: unable to connect to the SMTP server through TLS: tls: first record does not look like a TLS handshake” when testing the connection in Mattermost.