Yes, absolutely no problem. You will need separate directories and databases per instance as well as separate listening ports.
F.ex,:
Instance 1
Application directory: /opt/mattermost1
Database: mattermost1
Port: 8001
Instance 2
Application directory: /opt/mattermost2
Database: mattermost2
Port: 8002
And you will need one nginx in front of both instances that will reverseproxy the domain „mattermost1.com“ to „localhost:8001“ and „mattermost2.com“ to „localhost:8002“.