I’ve been thinking about this for a while and haven’t made a decision on what I want to do, so I decided to post a question to see how others are doing this, and to discuss some of the pros / cons of the different approaches.
Generally speaking most (including me) want the following out of their homelab at a high level:
Good performance
Their data to be safe
Not paying millions of dollars to the electric company to run it
I know I won’t be able to check every single box above, but what does everyone think about the following scenarios?
My general plan is to have a central NAS and a back-up NAS that will power up once daily to do backups then power back down, and 2x VM hosts (thin clients for the power savings)
I plan on using TrueNAS core for my NAS’ and XCP-NG for my VM Hosts (possibly proxmox if theres a strong reason as to why I would be better off)
VM Storage on central NAS
Performance can be fantastic if setup properly, however I don’t think I’m going to get NVMe level performance for my VM hosts unless I make the jump to 40GB network equipment (correct me if I’m wrong please)
Do I even need NVMe level performance? I know this depends on the work load and my goals in homelab, etc… but am I just expecting to much, or being to overkill?
Data will be safe, as I’m using ZFS with sync writes, etc…
VM Storage on VM host (NVMe drive in each VM host)
Performance here I think would likely be better than option 1 without the need for fancy expensive networking equipment as its local NVMe.
I would need 2 NVMe’s per host so that they can be mirrored for some sort of protection, or could I have the storage replicate to the other host? I know RAID isn’t a backup, so a mirrored array in each host would be for the up-time only if a drive failed, I would need to replicate or backup.
Would this effect the ability to migrate VM’s between the hosts (I assume this is much easier if the VM storage is in a central spot)
One of my use cases is Plex and the storage required for the media wouldn’t be able to go into a thin client VM Host, so I will need a NAS and it will need to be powered on pretty much 24-7, so I don’t know that theres a reason to move my VM storage off the central NAS and to the individual hosts, unless the performance gains make it worth it.
Don’t use thin clients, and use like an R730 with all my storage, vm’s etc… all on a single device
The main reason I don’t want to do this is cause its single point of failure, but the performance of local NVMe would be there, I could have ZFS pools for my bulk storage, and its less network equipment, NICS, etc… so it would be cheaper on the electric bill to run.
I would go with TureNAS SCALE because CORE is on life support. Performance is completely up to you. You’ll need to figure out the performance based on the raid you decided to use and vdev layout and cache drives. Personally I would at least put in an optane drive for the SLOG and turn on sync always on your iscsi/nfs dataset for your SR on xcpng.
Leading into your xcpng hosts - I wouldn’t bother stacking hard drives on each host. I would set up and iscsi/nfs target on your TrueNAS as I mentioned above. That way you have HA functionality. I currently use 2 HP 800 G3 i7’s with 32GB each. the current power draw with my supermicro (TureNAS) server is is about 300 watts. Supermicro is an old server that I plan to replace soon. I have game servers running, nextcloud instance, unifi controller and some vm’s I tinker with. All on 1Gb connection.
I used to use a r720 as my xcpng host and with both my supermicro and r720 with local ZFS I was pulling 500-550 watts from the wall. so really its up to you on what matters most. I think you can get comparable performance on your TrueNAS if you design your pool layout properly.
Recommendation:
1 liter PC’s for xcpng with a lot of RAM in a single pool for HA
TrueNAS as iscsi/NFS with optane drive for SLOG for SR
Flash or spinning rust in mirror’s for best performance
I feel like I’ve heard the “TrueNAS Core is going away” thing before but it never ends up being true. Granted its been a while, was there some sort of announcement that I missed?
I know originally TrueNAS Core performance was better than Scale if you were using it for pure “storage” purposes, Scale became attractive when you wanted to do VM’s, and containers, etc…
So I did some more up-to-date research and it seems like the writing is on the wall that TrueNAS core will be going away at some point.
I don’t think that will be tomorrow, but I may want to use Scale to ensure I’m supported going forward.
The only question that remains then is for core vs scale, is the performance pretty much equal now? I don’t mean the extra bells and whistles like jails, and apps and all that.
From the “core NAS” mindset, is a ZFS pool with an NFS share the same speed on Scale as it is on Core?
IXsystems is rolling out scale on production enterprise systems now. I believe they have tuned the software to be very similar in performance if not better in some cases on the linux kernel. At least from what I have used in production professionally and in my home lab both using scale and core.
I think you need to define these terms for us before anyone can make a reasoned recommendation. I think my homelab provides good performance, I think my data is safe, and all in my home lab consumes about 120 watts of power including everything, even dual WAN modems. But you may not be happy with the same level of performance that I am happy with. I run two Proxmox hosts, one is a 35 watt TDP HP Elitedesk Mini pc with an intel i5-12500T, 64 gb of memory and a pair of NVMes in a mirror. That is my main server hosting my websites (wordpress), nextcloud, and a bunch of services for home use (Home Assistant, Grocy, Mealie, Photoprism, etc.). My second Proxmox host is an N100 NUC style PC, that mainly acts a virtualized NAS to host my second copies of my backups and it runs Ansible to keep my environment up to date. I have a Synology NAS that stores all my data. I back up my VMs there, I keep my persistent docker volumes there, etc. My VMs all run on the NVMe drives in the HP mini PC. I can wipe my Proxmox machine, do a fresh install and restore my VMs from backup in approximately 30 minutes, so I don’t bother backing up my proxmox hosts at all. My websites back up every 2 hours to the Synology, so at most I lose 2 hours of data, which is negligible for me. I don’t make any money on my websites. Synology backs up my data to the N100 machine and to AWS Glacier, so I always have 3 copies of everything. My firewall is a pfSense fanless mini PC (quotom I believe), and I have a 2.5gb/10gb managed switch, but most everything is 2.5gb networking at the moment. I rely on Cloudflare tunnels a lot to expose my services and I use the cloudflare CDN/caching to speed up my website performance. I run the following devices for a total of 120 watts of power draw: 2 proxmox hosts, pfsense box, 30 port switch, synology NAS, a cable modem, a T-Mobile 5G home internet modem (I have redundant WAN connections), wireless access point, Ring alarm base unit, and a raspberry Pi 3 that is converted into a ham radio repeater (PiStar)
Anyway, I am not sure I would ever put VM storage on a NAS. If your data sits on the NAS, the VMs themselves take up very little room on your server. Also your second NAS can be as simple as a raspberry pi running Debian serving up a NFS share or acting as a RSYNC host. You don’t need a lot of performance for your back up system in my experience. I also don’t make any attempt to provide high availability. I am OK with a single point of failure so long as my data is safe.
There are certain parts of XCP-NG where you will not get performance! For your lab this might not be an issue.
The one area where performance is lacking is the migration from one storage to another storage. Having 10gbps and an array fast enough to handle 6gbps+ in benchmarks over NFS is till going to give you an issue when doing this type of migration, same goes for local to NAS migrations. In all my testing, a little over 400mbps is all I can get (still chasing this). Now that said, under certain conditions, I’m able to get nearly 7gbps during a benchmark using files that are double the cache RAM and on bigger systems, read them back ad cache ram speeds over NFS. There are several threads about this on the XCP-NG forums.
As far as using a NAS for storage, unless I’m wrong, you gain the ability to use the rolling pool upgrade because the system will migrate the ram portion of the VM to a second/third host, do the upgrade on the controller (was called master), then when that reboots, move them all to other hosts again and continue upgrading the rest of the hosts.
if you want to fool with HA, then network storage is a must have.
All that said, I look forward to the day when someone smarter than me documents the process of getting something like Ceph working for converged storage and I can ditch the the NAS (maybe).
As far as NAS goes, you will probably want to build your own, here’s a recent threard I made on my journey from 100 watts at idle down to 32 watts at idle (less with SSD if you need to buy drives).
I had a lot of leftover parts that went into this, and if you are willing to order from Ali Express, the main board was a lot cheaper there, or you can buy the n305 version to have a little more horsepower. I have not fully tested the limits of this little NAS, but I’m going to be switching things over soon to shut down the 100 watt idle NAS.
Many useful inputs have already been provided. I just add a few aspects you may want to consider:
you want to save on power? then do not consider 40Gbps network equipment. Even 10Gbps network equipment is more power hungry. Also: unless your VM vdsisk are really large, consider going for NVMEs only. Even if you consider putting VM vdisks on a NAS, consider just using NVMEs on the NAS; no HDDs. Prices are better these days, less power consumption, higher speed. But then you will want 10Gbps NICs, and use fiber instead of copper.
if you don’t need HA and don’t really need a second hypervisor, just use local NVME storage. If you want a backup hypervisor or HA, use the same hardware for both to allow for live migration and keep things simple.
TrueNAS / ZFS is great for performance for VM storage backend. For this you don’t need insame amounts of space, so keep this on NVME, as TrueNAS does not put idle HDDs to sleep.
TrueNAS Scale Electric Eel turned out to have bad (network?) performance on C3758 Atom CPUs, while TrueNAS Core performs well. Possibly a Linux vs FreeBSD driver issue on Denverton with some 10Gbps chipsets, just a guess. So the CPU/board / NIC combo may make a difference, you want to choose something where Scale runs well.
unRAID is great for large amounts of storage and flexibility, but speed is not great. It does put idle HDDs to sleep to save power. It is more economical than TrueNAS to use for lots of media data. For media data you don’t need more than 2.5Gbps, 1Gbps is even enough, unless you do video processing.
So I see 2 options:
(a) one hypervisor with local NVME storage for all vdisks with 2.5Gbps, plus one unRAID with HDDs and NVME cache and 2.5Gbps
(b) idenical hypervisors with 10Gbps, one TrueNAS Scale with NVMEs and 10Gbps, one unRAID with HDDs and NVME cache and 2.5Gbps