I reverted to bare metal setup for the same reason. I didn’t see any known vulnerabilities, but the possibility of it made me nervous. Plus I had an old Dell SC1525 1U server that wasn’t good for much else. But if I was in your situation @Fortress it would be worth the trouble to avoid wasting the resources. Unless you’re a target and have reason to fear baddies actively working to break your network security, you should be fine using a virtualized edge router. Whatever type of hypervisor you go with, you should keep an eye out for any security vulnerabilities that are announced.
BTW, have you looked into type 1 hypervisors like XCP-ng or ESXi? The guest OS’es are a bit more secure since they don’t share a host OS.