I’m not familiar with the vultr setup, sorry - but these kind of problems usually relate to so-called hairpin or uturn configurations, that is when you try to connect from the inside of a server to its outside IP.
Is the public IP the domain points to directly available on your server or is there any kind of public firewall or NAT in front of it? You can check if you have the IP directly by using the command ip a on the console of your linux host, if you can see the public IP your subdomain.mydomain.com points to, it means yes.
Next thing you can try is to connect to the public IP directly on the linux host (not from inside the docker container) - does this work?