Storage Layout Suggestions for New XCP-NG / TrueNAS Scale Home Server

Computer Hardware & Server Infrastructure Builds
1

Hi everyone. First post.

Looking for some input on this personal project as I’ve never used or installed xcp-ng, nor TrueNAS (Scale) – this will be my first venture using both. I’ve been acquiring hardware for some time to build a new home server that will host a few general purpose linux VMs with one of those VMs being TrueNAS Scale for NAS duties. Plan is for a single xcp-ng host with local storage, no clusters. I’m currently stuck a bit on the local storage design / layout in terms of what would be optimal.

Server Hardware

  • Supermicro SC846 4U case with the “A” backplane & “quiet” power supplies.
  • ASUS Pro WS W680-ACE IPMI Motherboard
  • Intel i7-13700k CPU
  • 128GB DDR5 ECC RAM
  • LSI 9300-8i HBA with (8) 8TB WD Red drives
  • (3) 2TB WD SN850X NVME drives that will fill the MB’s (3) M.2 slots
  • ConnectX-5 25GbE dual port NIC

I plan to pass through the LSI HBA and (8) disks to the TrueNAS Scale VM.
The debate is how I should setup the NVME drives. Should I:

  1. Setup the (3) NVMEs as local storage to the xcp-ng host, use them as the xcp-ng boot/OS drive and SR. Concern is I’d like to be able to use this storage pool as fast NAS storage for clients on the network, so if I did this, I’d want to share out some of this capacity to the network. 2TB is a lot for just xcp-ng boot and SRs. Also not sure what local storage options would work for 3 drives other than ZFS which seems to overly complicate the xcp-ng local storage. Could just use 2 and mirror if I go this way. Not sure how I’d share out a part of this local NVME space short of installing Samba or something on the xcp-ng host which sounds like a bad idea.
  2. Go buy a couple small/cheap SATA SSDs to act as the xcp-ng boot drive & SR. Connect them to the MB’s onboard SATA controller. Pass the NVME drives over to the TrueNAS Scale VM and setup the (3) NVME drives as a 2nd “fast” storage pool. Only downside other than cost and complexity is my VMs wouldn’t be hosted on NVME, only on the SATA SSDs – not sure that would be a noticable difference.

Favoring option #2.

Thoughts?

2

I am not a big fan of virtualizing TrueNAS but of the two options #2 sound better.

3

Thanks for the response Tom.

If this was your sole home server, would you rather install TrueNAS Scale on bare iron and run the ~6 linux VMs & few Docker containers natively on Scale as the hypervisor? One of the linux VMs was going to be Docker host. I wasn’t sure Scale’s virtualization and container hosting feature maturity was quite there yet.

A lot of people seem to prefer the xcp-ng (or proxmox) on bare iron so that you can snapshot TrueNAS before upgrades, and that they have better virtualization support than Scale.

Again, this my first venture into both xcp-ng and Scale, so going in a bit blindly.

4

You can just run TrueNAS and have a single VM that runs all the docker things.