Configuring TURN Server for Mattermost Calls

Hi all,

I just upgraded from 6.3.9 ESR to 7.1.2 ESR and want to give Calls a test ride.

However, I also from the beginning want to configure a TURN server to avoid one way connection issues.

I tried to add it to the corresponding config section in the plugin’s setup screen, but afterwards the plugin does not start any more and complains about an invalid URL…

Unfortunately, neither the Mattermost documentation, nor the Calls plugin’s GitHub page, not Google were able to give any explanation about how exactly the settings are supposed to be formatted…

[{"urls":["stun:turnserver.example.org:3478", "turn:turnserver.example.org:3478"]}]

invalid ICEServers value: URL is not a valid STUN/TURN server

Did anyone already figure it out?

Hi @GOhrner ,

the plugin is still in beta and so is the documentation :slight_smile: The key to make that work is to provide a different array per protocol, as outline here:

[
   {
      "urls":[
         "stun:turnsrv01.xxx.yyy.de:3478"
      ]
   },
   {
      "urls":[
         "turn:turnsrv01.xxx.yyy.de:443?transport=udp",
         "turn:turnsrv01.xxx.yyy.de:443?transport=tcp"
      ]
   }
]

Can you please check if that works for you?

This way the plugin at least starts again.

I wasn’t yet able to actually test the Call feature itself with other partys, so I don’t know if it actually works this way.

Also, I still don’t really understand this data structure… It’s a list of dicts of lists of URL strings - why not just take a plain list of URL strings in the first place and get rid of the remaining structure?
I’d have understood it if the individual URL lists would be tagged as “stun” or “turn” URLs, but they aren’t and the server’s type is indicated by the URL anyway…

Since this plugin is still in beta and under heavy development, I think it’s OK to experiment a bit with the configuration options necessary as well as the format to use them internally then. As soon as the plugin goes GA, I’m sure there will be better UI controls and proper documentation to help the users better understand the configuration options.

Let me know if you were able to make a call with these settings, there might be some other tunings necessary if your calls drop after a few seconds (websocket timeouts, etc.).

Ok, no worries - let’s see how it develops.

I have misinterpreted the MM 7.0 announcement, which clearly says “beta”. Let’s try to use it and provide feedback

Unfortunately, it doesn’t work for me at all, and the server log error messages are not very helpful for me to try to track down the cause.

I can initiate a call, but once the other participant tries to join (s)he is thrown out again after less than a second without an error message on their side.

The server log says the following (this results from multiple attempts to join the conversation / call):

{"timestamp":"2022-08-21 18:04:53.788 +02:00","level":"error","msg":"callback failed: not found","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*Plugin).handleLeave websocket.go:372"}
{"timestamp":"2022-08-21 18:05:10.245 +02:00","level":"error","msg":"callback failed: not found","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*Plugin).handleLeave websocket.go:372"}
{"timestamp":"2022-08-21 18:05:22.372 +02:00","level":"error","msg":"callback failed: not found","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*Plugin).handleLeave websocket.go:372"}
{"timestamp":"2022-08-21 18:05:25.223 +02:00","level":"error","msg":"callback failed: not found","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*Plugin).handleLeave websocket.go:372"}
{"timestamp":"2022-08-21 18:05:29.230 +02:00","level":"error","msg":"callback failed: not found","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*Plugin).handleLeave websocket.go:372"}
{"timestamp":"2022-08-21 18:05:38.353 +02:00","level":"error","msg":"callback failed: not found","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*Plugin).handleLeave websocket.go:372"}
{"timestamp":"2022-08-21 18:05:51.916 +02:00","level":"error","msg":"callback failed: not found","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*Plugin).handleLeave websocket.go:372"}

Any ideas? Is this still a TURN server config issue, or some internal Plugin / Server incompatibility, or some issue regarding web connectivity?

My Mattermost runs behind an Apache reverse proxy, so maybe I need to pass through additional ports or do some other config setting on the reverse proxy side?

I looked for info regarding this, but couldn’t find any in the Calls or Mattermost reverse proxy documentation.

The RTC server seems to listen globally, though. netstat says:

udp        0      0 0.0.0.0:8443            0.0.0.0:*                           1318/plugins/com.ma 

I’m not sure if this is related to TURN server, but the error message suggests that it has something to do with the websockets, but that could also be unrelated or an inherited error.
Not sure if your firewall would allow that, but the RTCd port (udp/8443) needs to be directly accessible by the clients and using SSL for your Mattermost URL is also a requirement for Calls to work.

To rule out that your STUN and/or TURN configuration is causing issues or not working as expected, could you please try to remove the configuration again and only use the ICE Host Override setting, pointing to your Mattermost domain (without the protocol scheme, so just mattermost.yourdomain.com), restart the calls plugin (disable, enable) and test the plugin again, please?

With this configuration, it sometimes worked and sometimes didn’t.

I.e. in some cases, the call was terminated immediately / not really established, while in other cases it worked ok.

The colleague with which I tested was in the same office, though, so this probably wasn’t a very realistic test.

Some peculiar thing I noticed with screensharing: My colleague first used the Mattermost Desktop application in Windows, and in this case it shared all of her screens immediately without any confirmation.

On the other hand, accessing the Mattermost web page in a browser bringst up a dialog after clicking the share button in which you can select the screen or even Window you like to share.

Calls never seem to work while our company VPN is active (OpenVPN with split routing, IPv4 goes though the tunnel, IPv6 doesn’t), so we tested everything without VPN. Without VPN the behaviour described above happens - sometimes, calls can be established, sometimes not. (e.g. just clicking rejoin after leaving a call then fails).

Here’s the last few lines of the server log with user and session IDs obfuscated, but I kept same strings the same after obfuscation.

BTW which of these IDs are sensitive and which can just be pasted here verbatim without security implications?

{"timestamp":"2022-08-23 19:04:09.820 +02:00","level":"warn","msg":"Unrecognized config permissions tag value.","caller":"api4/config.go:426","tag_value":"sysconsole_write_*_read"}
{"timestamp":"2022-08-23 19:04:09.830 +02:00","level":"info","msg":"rtc: server was shutdown","caller":"app/plugin_api.go:937","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Info log.go:84"}
{"timestamp":"2022-08-23 19:04:09.838 +02:00","level":"info","msg":"plugin process exited","caller":"plugin/hclog_adapter.go:61","plugin_id":"com.mattermost.calls","wrapped_extras":"pathplugins/com.mattermost.calls/server/dist/plugin-linux-amd64pid28855"}
{"timestamp":"2022-08-23 19:04:13.785 +02:00","level":"warn","msg":"Unrecognized config permissions tag value.","caller":"api4/config.go:426","tag_value":"sysconsole_write_*_read"}
{"timestamp":"2022-08-23 19:04:30.818 +02:00","level":"warn","msg":"Unrecognized config permissions tag value.","caller":"api4/config.go:426","tag_value":"sysconsole_write_*_read"}
{"timestamp":"2022-08-23 19:04:30.895 +02:00","level":"info","msg":"rtc: server is listening on udp 8443","caller":"app/plugin_api.go:937","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Info log.go:84"}
{"timestamp":"2022-08-23 19:04:30.896 +02:00","level":"info","msg":"rtc: server is listening on udp 8443","caller":"app/plugin_api.go:937","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Info log.go:84"}
{"timestamp":"2022-08-23 19:04:30.896 +02:00","level":"info","msg":"rtc: server is listening on udp 8443","caller":"app/plugin_api.go:937","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Info log.go:84"}
{"timestamp":"2022-08-23 19:04:30.896 +02:00","level":"info","msg":"rtc: server is listening on udp 8443","caller":"app/plugin_api.go:937","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Info log.go:84"}
{"timestamp":"2022-08-23 19:04:30.897 +02:00","level":"info","msg":"rtc: server is listening on udp 8443","caller":"app/plugin_api.go:937","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Info log.go:84"}
{"timestamp":"2022-08-23 19:04:30.897 +02:00","level":"info","msg":"rtc: server is listening on udp 8443","caller":"app/plugin_api.go:937","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Info log.go:84"}
{"timestamp":"2022-08-23 19:06:16.595 +02:00","level":"info","msg":"pc ERROR: 2022/08/23 19:06:16 Incoming unhandled RTP ssrc(1693057650), OnTrack will not be fired. mid RTP Extensions required for Simulcast\n","caller":"io/io.go:428","plugin_id":"com.mattermost.calls","source":"plugin_stdout"}
{"timestamp":"2022-08-23 19:06:34.402 +02:00","level":"error","msg":"failed to read RTP packet","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92","error":"EOF"}
{"timestamp":"2022-08-23 19:07:04.481 +02:00","level":"error","msg":"screenSession should not be nil","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92"}
{"timestamp":"2022-08-23 19:07:18.231 +02:00","level":"error","msg":"failed to read RTP packet","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92","error":"EOF"}
{"timestamp":"2022-08-23 19:07:22.010 +02:00","level":"error","msg":"failed to read RTP packet","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92","error":"EOF"}
{"timestamp":"2022-08-23 19:07:22.010 +02:00","level":"error","msg":"failed to read RTP packet","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92","error":"EOF"}
{"timestamp":"2022-08-23 19:07:46.234 +02:00","level":"error","msg":"failed to read RTP packet","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92","error":"EOF"}
{"timestamp":"2022-08-23 19:08:38.046 +02:00","level":"error","msg":"screenSession should not be nil","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92"}
{"timestamp":"2022-08-23 19:08:45.307 +02:00","level":"error","msg":"failed to read RTP packet","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92","error":"EOF"}
{"timestamp":"2022-08-23 19:08:45.307 +02:00","level":"error","msg":"failed to read RTP packet","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92","error":"EOF"}
{"timestamp":"2022-08-23 19:08:51.004 +02:00","level":"error","msg":"failed to read RTCP packet","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92","error":"io: read/write on closed pipe"}
{"timestamp":"2022-08-23 19:08:59.545 +02:00","level":"error","msg":"failed to read RTCP packet","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92","error":"io: read/write on closed pipe"}
{"timestamp":"2022-08-23 19:09:00.987 +02:00","level":"error","msg":"failed to add screen track","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92","error":"timed out signaling","sessionID":"session_id_1"}
{"timestamp":"2022-08-23 19:09:09.480 +02:00","level":"error","msg":"failed to add screen track","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92","error":"timed out signaling","sessionID":"session_id_2"}
{"timestamp":"2022-08-23 19:09:09.680 +02:00","level":"error","msg":"failed to read RTP packet","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92","error":"EOF"}
{"timestamp":"2022-08-23 19:09:09.680 +02:00","level":"error","msg":"failed to read RTP packet","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92","error":"EOF"}
{"timestamp":"2022-08-23 19:09:37.020 +02:00","level":"error","msg":"screenSession should not be nil","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92"}
{"timestamp":"2022-08-23 19:09:56.837 +02:00","level":"error","msg":"screenSession should not be nil","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92"}
{"timestamp":"2022-08-23 19:10:09.663 +02:00","level":"error","msg":"failed to read RTP packet","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92","error":"EOF"}
{"timestamp":"2022-08-23 19:10:09.663 +02:00","level":"error","msg":"failed to read RTP packet","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92","error":"EOF"}
{"timestamp":"2022-08-23 19:10:51.622 +02:00","level":"error","msg":"failed to read RTCP packet","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92","error":"io: read/write on closed pipe"}
{"timestamp":"2022-08-23 19:11:01.528 +02:00","level":"error","msg":"failed to add screen track","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92","error":"timed out signaling","sessionID":"session_id_3"}
{"timestamp":"2022-08-23 19:11:33.449 +02:00","level":"warn","msg":"websocket.slow: dropping message","caller":"app/web_conn.go:716","user_id":"user_id_1","type":"channel_viewed"}
{"timestamp":"2022-08-23 19:11:33.897 +02:00","level":"warn","msg":"websocket.slow: dropping message","caller":"app/web_conn.go:716","user_id":"user_id_1","type":"channel_viewed"}
{"timestamp":"2022-08-23 19:12:02.757 +02:00","level":"warn","msg":"websocket.slow: dropping message","caller":"app/web_conn.go:716","user_id":"user_id_1","type":"channel_viewed"}
{"timestamp":"2022-08-23 19:12:09.170 +02:00","level":"error","msg":"failed to read RTCP packet","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92","error":"EOF"}
{"timestamp":"2022-08-23 19:12:09.170 +02:00","level":"error","msg":"failed to read RTP packet","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92","error":"EOF"}
{"timestamp":"2022-08-23 19:12:09.170 +02:00","level":"error","msg":"failed to read RTP packet","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92","error":"EOF"}
{"timestamp":"2022-08-23 19:12:17.488 +02:00","level":"error","msg":"failed to read RTP packet","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92","error":"EOF"}
{"timestamp":"2022-08-23 19:12:17.488 +02:00","level":"error","msg":"failed to read RTP packet","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92","error":"EOF"}
{"timestamp":"2022-08-23 19:12:17.488 +02:00","level":"error","msg":"failed to read RTCP packet","caller":"app/plugin_api.go:940","plugin_id":"com.mattermost.calls","origin":"main.(*logger).Error log.go:92","error":"EOF"}
{"timestamp":"2022-08-23 19:12:17.695 +02:00","level":"warn","msg":"websocket.slow: dropping message","caller":"app/web_conn.go:716","user_id":"user_id_1","type":"channel_viewed"}
{"timestamp":"2022-08-23 19:12:17.695 +02:00","level":"warn","msg":"websocket.slow: dropping message","caller":"app/web_conn.go:716","user_id":"user_id_1","type":"channel_viewed"}
{"timestamp":"2022-08-23 19:12:17.842 +02:00","level":"warn","msg":"websocket.slow: dropping message","caller":"app/web_conn.go:716","user_id":"user_id_1","type":"channel_viewed"}
{"timestamp":"2022-08-23 19:12:17.842 +02:00","level":"warn","msg":"websocket.slow: dropping message","caller":"app/web_conn.go:716","user_id":"user_id_1","type":"channel_viewed"}
{"timestamp":"2022-08-23 19:14:32.957 +02:00","level":"info","msg":"SimpleWorker: Job is complete","caller":"jobs/base_workers.go:88","worker":"ExpiryNotify","job_id":"xxxxxxxxxxxx"}

With which one? The one you had or the one I suggested?

If it doesn’t work when sitting next to each other, I think you do not need to create greater distances :slight_smile:

This has been fixed in the desktop app >= 5.1, which supports for selecting the screen or application to share after clicking on the share button. So your colleague probably still has an older version of the app which would need to be updated now in order to properly support Calls.
5.1 has many other improvements (including an auto-updater), so it’s worth upgrading to it anyways.

Is your mattermost instance reachable via IPv6 internally? Is the mattermost server also reachable without VPN and if so, do the public DNS records also return an AAAA record for it?

Your logs definitely show some network errors here, so let’s try to get things stable while in your internal network first.

Ok, I’ll try to clarify:

Yesterday, I only tested your new proposal:

  1. I removed the previous stun/turn server configuration, so this config text box was empty.
  2. I entered my Mattermost host name (without protocol part and without port specifier) into ICE Host Override.
  3. I stopped and re-started the plugin.

After it didn’t work on the first attempt when my colleague still had VPN enabled, we tried without VPN. However, there may be traces from attempts with VPN in the server logs.

Mattermost is reachable without VPN - we need VPN to access other internal IPv4 based services.

The reverse proxy in front of Mattermost is also reachable through both IPv4 and IPv6.

Mattermost itself was only listening on 127.0.0.1, i.e. IPv4 localhost, behind the reverse proxy. I now changed this to ::1 / IPv6 localhost, but don’t expect this to make a difference.

As far as I can interpret netstat, the Calls plugin seems to listen to IPv4 UDP only on all interfaces, before and now:

udp        0      0 0.0.0.0:8443            0.0.0.0:*                           25297/plugins/com.m 
udp        0      0 0.0.0.0:8443            0.0.0.0:*                           25297/plugins/com.m 
udp        0      0 0.0.0.0:8443            0.0.0.0:*                           25297/plugins/com.m 
udp        0      0 0.0.0.0:8443            0.0.0.0:*                           25297/plugins/com.m 
udp        0      0 0.0.0.0:8443            0.0.0.0:*                           25297/plugins/com.m 
udp        0      0 0.0.0.0:8443            0.0.0.0:*                           25297/plugins/com.m 

I’d have expected it to listen to [::]:8443, at least additionally, but that does not seem to be the case.

Host OS is still Debian 10 Buster, upgrade to Bullseye will happen soon.

It doesn’t, since it’s shielded by the reverse proxy.

Good catch - I’ve brought that up with the developers.
With regards to your call drops: Can you please have the browser or application console open while running the call so we can see if there are any relevant error messages logged on the clientside?
Also during a call, in the channel, where you started the call, you can type /call stats to get statistics about the current situation, which might also be helpful for debugging.

Just one addition after having talked with the developers of this plugin:

The official documentation has already been updated, although the plugin is in beta (see Configuration settings — Mattermost documentation)

We could also try to diagnose the connection by running a netcat server on your mattermost host on a different port (f.ex. nc -vlnup 8444) and try to connect to it from the clients to simulate the calls plugin and see if that works at least. It would also be interesting to compare the results of this test for connections with and without an established VPN. The clients could use nc -vu mattermost.server.name 8444 and it should be possible to send data in both directions. Here’s what it should look like:

# server
$ nc -vnlup 8444
listening on [any] 8444 ...
connect to [10.192.66.10] from (UNKNOWN) [10.0.2.140] 59590
test from client
test from server

# client
# nc -vu 192.168.0.1 8444
192.168.0.1: inverse host lookup failed: Unknown host
(UNKNOWN) [192.168.0.1] 8444 (?) open
test from client
test from server

Ideally, you would use the real port 8443 as well as the domainname of your mattermost server for the tests. To do that, you can disable the calls plugin, so port 8443 is free again.

IPv6 is not supported currently for the calls plugin, since the development is in an early stage and the underlying framework has not been tested on IPv6 as of yet. There will definitely be IPv6 support later on, but for now, it’s v4 only.

Can this be the issue? The server is reachable by IPv6, and if the browser tries to connect but Calls doesn’t listen on IPv6, the packets will just be dropped. Or is there any mechanism in the WebRTC standard which mandates the browser to retry with IPv4 if IPv6 fails?

That’s exactly what happened when I tried the nc commands you proposed:

There’s no packet filter between client and server.

However netcat-traditional - which I used on the server side - also only supports IPv4.

netcat from OpenBSD, which I used on client side (I accidentally used different packages at first and noticed when analyzing what happened), on the contrary, does support IPv6.

This caused the connection to fail because the nc server only listened on IPv4 and the nc client tried to connect using IPv6.

It only works if I force the client to IPv4 by using the -4 switch or using the numeric IPv4 address instead of the hostname.

So if that’s the reason, I would have to remove the AAAA record for the whole subdomain used by Mattermost - or am I mistaken?

Yes, this could very well be the reason and I also thought about that, but it does not make sense with regards to your comment “Calls never seem to work while our company VPN is active”.
If they work when the VPN is not active, then there’s a higher chance that IPv6 is being used…

Instead of removing the AAAA record for your mattermost subdomain, could you try to specify the IPv4 address of the Mattermost server in the ICEHostOverride setting or create a new record (mattermost-calls.yourdomain.com?) with just an A record and use that in the ICEHostOverride option to see if that helps?