I’ve been thinking about this lateley, im some what fimiliar with both FreeNAS and PfSense but i’ve only labbed with it in my home virtual enviroment (hyper-v) which is no way close to an real production enviroment.
Let’s assume i have a costumer and they have bought a server from me, i want to run both FreeNAS as a storage solution and PfSense as a Routing solution.
They only backside that i can think of is that if i have to reboot the host the internet does down.
Which may affect some of the other VM’s or the uptime in production.
Please correct me if im wrong.
FreeNAS (ZFS Specifically) relies on direct access to the hard drives. While you can technically pass the hard drives through individually in most hypervisors, the only supported (and its barely supported at that) method is to attach the hard drives to a separate controller, of the kind that works with ZFS without virtualization being added, and pass it through to the VM as a PCIe device. This means that the hard drives attached to this controller can’t be seen by the OS at all. Therefore, you can’t use those drives to store VMs. Also, it is NOT supported, and a very bad idea, to mount a share from FreeNAS via NFS, iSCSI, or any other method, in the host OS in order to store VMs or backups. If you mount the FreeNAS storage in other VMs, make sure that you set up the automatic boot order so that FreeNAS comes before them, and with a timer of perhaps 5 minutes before the other VMs boot (depending on the normal boot time of your FreeNAS).
PFSense works great as virtualized, with the caveat as you mentioned that the internet will go down, however on some hypervisors you have to disable the hardware checksum offloading, and on all hypervisors it helps to select the correct type of virtual NIC. See here for details: https://docs.netgate.com/pfsense/en/latest/virtualization/index.html
In a production environment run pf-Sense on bare metal or an appliance from Netgate. Same for FreeNAS since ZFS wants to see raw devices, it is both device and file manager. Go to the respective web sites read the docs and blog posts. Lots of excellent videos on youtube as well.
Only problem is that where i live (Sweden) we don’t really have a Netgate reseller… As far as i know everyone here seems to work it virtualized. Not 2 sure though.
If I was going to run both FreeNAS and pfSense in VMs on the same server, I would set it up where the SAS controller was passed thru to the to the FreeNAS box, also I would pass-thru two ethernet ports to the pfSense - with the assumption that you have a 4 port NIC installed. I would want the pfSense to be isolated from the rest of the VMs networkly on the server, and direct access since it is a firewall. Once you have it setup, you could put your VMs on a FreeNAS iSCSI, but you need to ensure that your FreeNAS startups first and the rest of you VMs are delayed until the iSCSI share has started. But this is not a very good solution, it will work, but not something I would want mission critical VMs on.