For 1 one has to note that if both Mattermost and Nextcloud share the domain embedding Mattermost with an iframe will work out of the box (but you will miss browser notifications since they generally do not work through an iframe).
What I mean with “share the domain” is Mattermost being available via your-server.domain.com/mattermost and Nextcloud being available via your-server.domain.com/nextcloud.
A way around the missing notifications would be https://github.com/Kopano-dev/mattermost-plugin-notifymatters which implements a post message api in Mattermost, but then afaik the “external site” plugin still needs to understand how to use this api.
In case Mattermost and Nextcloud are running on different (sub) domains you need to modify the csp through your vhost to make it possible to load Mattermost in an iframe from a different domain.
edit: for 2 your best choice is probably to find an external authentication source that works with both Nextcloud and Mattermost (like LDAP, or possibly Gitlabs Oauth)