FreeNAS Storage Idea (Possible HA)

I am thinking about moving away from my current storage solution to a FreeNAS solution and wanted to get some feedback as I am new to FreeNAS and the ZFS file system. I am wanting to get as close as I can to a solution that would be considered to have High Availability (HA) and also a few or no “single points of failure” (SPOF) to minimize downtime when it comes to my storage solution. I have a small home business where I do IT consulting and have recently been asked if I could support the hosting of websites and possibly email services.

Right now my limitations on my current setup other than the storage solution when it comes to HA and SPOF is that I only have 1 power source into my location (though I do have a number of UPSs installed for battery backup) and also that I only have 1 internet connection. When it comes to storage this is a different story and hence the desire to implement a better solution.

I have been using Proxmox for a few years now to run the various servers/services I have on my network and have been using QNAP NAS boxes for my storage. Right now I run 3 QNAP boxes, 1 for personal storage and VM backups, the second for VM storage for my Proxmox Nodes (I have 4 nodes currently) and the other as shared storage for my Nextcloud install and to temporarily store downloaded files as well as other random storage needs.

I am wondering which is the better way to go for my eventual move to FreeNAS as my new storage solution and this is where my question sort of splits in two. The first is hardware-based and here are the ideas I have had so far in terms of possible solutions for building 2 or 3 servers:

  1. Using other enterprise-level servers such as the Dell R720XD.

  2. Building systems from new parts though less enterprise-level hardware due to cost. Though they would need to have the following key features: Rack-mountable 2U or 4U chassis, 12 hot-swappable data drive bays, 2 hot-swappable OS drive bays, motherboard IPMI, redundant power supplies and at least 4 or 5 PCIe expansion slots.

The second is when it comes to installing FreeNAS, I had the following ideas and I would love feedback from the community on its viability and also on what would be the best storage solution for my needs.

  1. Setup FreeNAS on bare metal hardware and use multiple servers and data replication between them to help with hardware failure and also data preservation. Though it would require manual intervention if there was an issue before services could be restored to normal operation.

  2. knowing that FreeNAS does not support HA but was wondering if the following would be possible to give FreeNAS HA like feature(s) and be a suitable production setup. The idea would be to virtualize FreeNAS in say a 3 node Proxmox Cluster using that would be more of a head server configuration and then use 2, 3 or even 4 DAS or JBOD servers for the data drives.

Here is how I would see that solution working. By virtualizing FreeNAS under Proxmox and should the node FreeNAS is operating on have a failure of some kind the HA features of Proxmox would move FreeNAS to another node automatically. This, of course, does not address the storage aspect when it comes to the data drives and here is what I am thinking for that. If the multiple DAS/JBOD servers are attached to each Proxmox Node and all nodes are identical could I not use PCIe passthrough to FreeNAS. This would give FreeNAS access to the various disks for not only data storage but also data replication between the various DAS/JBOD chassis which would provide protection from a hardware failure when it comes to data storage.

Of course, if the Proxmox node were to fail and FreeNAS was moved to another node they would lose connection to the storage disks if they were only connected to the one node. What if each Proxmox node was connected to each DAS/JBOD server in such a way that it would not matter which node was hosting FreeNAS that FreeNAS would always have access to the storage disks. I know on Proxmox when It comes to network interfaces you are able to create various network bridges with the same identifier and as long as the VM that is moving to the new node as access to the same bridges with the same name they can run on a totally different node and network interface card.

I do not know if it is possible on Proxmox to even another hypervisor to create a virtual HBA while still using PCIe passthrough to an actual HBA card that would allow for this solution to work.

I know that it would obviously be more hardware than just running a FreeNAS box on bare metal but I am curious if this is possible and also if I could then extend this storage solution when the time comes by adding more DAS/JBOD servers and then expanding the existing FreeNAS setup or creating another FreeNAS install using the same Proxmox Nodes to host the FreeNAS install(s).

FreeNAS is great, but you are correct on the HA for storage being a single point of failure. That is why they have the TrueNAS line of serves that have redundant motherboards to mitigate that risk.

I am not sure how / if this works with Proxmox but with Xen Orchestra / XCP-NG you can setup a fast VM disaster recovery solution using their continuous replication.

1 Like

Tom, I had a look at both of those videos and wanted to say thanks for not only for your reply but also your YouTube channel, which has taught me many things including pfSense and also Unifi (recently moved from Cisco WAP150 to NanoHD) but I don’t think that your VM Disaster Recovery is the right solution. I did look into TruNAS and has a phone meeting with them but they are cost-prohibitive for me.

Do you know if on XCP-NG if there is a way to pass through either HDD/SSDs or an HBA or other PCIe card in such as way as they are transparent to the virtual machine using them?

What I am meaning is could you make it so like a network bridge I could have different HBA cards in the hypervisor nodes and the VM would use them and see them as all the same card, much like what happens with a virtual network interface.

In super simplified terms could I have a JBOD server that is connected to more than one hypervisor node and pass it through in such a way that no matter which node FreeNAS is running on it sees the drives the same way? Thus allowing the FreeNAS VM to work on any of the clustered nodes and still work as if the JBOD server was directly attached to the FreeNAS install?

You can do passthrough with XCP-NG, but it is not supported through the UI.