I would like to pick the brains of all the TrueNAS Scale enthusiasts on this forum. I have a Dell PowerEdge R730xd with 384 GB of RAM and Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz. I’m setting this up for my church and I’m trying to figure out whether to use all 26 drive bays (front 24 + the 2 in the rear) or just the 24 in the front and use the 2 rear as mirrored boot discs. I’ve already installed TrueNAS unto a USB flashdrive plugged into the internal USB port, but I saw a very strong warning against doing this in the UI upon initial login, so now I’m rethinking that.
Additionally, I’ve gathered from watching Tom’s videos that he seems to recommend no more than 10 discs per VDEV. I tried to use the online RAIDZ Capacity Calculator on the TrueNAS website, but it was quite honestly, very confusing.
So, I’m reaching out to this community to see if anyone is willing to help me figure out the best path forward. Thanks.
Hey, sounds like an awesome project. I’d stick with the mirrored boot pool and use the front bays for storage. Also, think through your workload before jumping in. I say that because striped mirrors are much much more performant than RAIDZ pools, so it may be worth your while to have maybe a four or five-vdev-wide mirrored pool alongside a bigger bulk storage RAIDZ. The striped mirrors would be great for things like running VMs and then the RAIDZ could be used for file sharing and backups.
That’s my two cents, but there is, of course, no right way.
I’m most certainly appreciative of your two cents, so thanks for contributing it to my offering bowl of ideas. Yeah, I’m very excited to get this project completed - it’s been sitting on my to-do list for a few months now.
So concerning my use case. Well, my desire is to use this as the shared storage backend for my XCP-ng pool, and also use it as a network share for a small group of Windows, Linux, and some Mac users (very light use - mostly reads from the media team on weekends). We do mostly livestreams of our services, and then store the footage for archival purposes. All video editing (if any) is done on local machines, so no need to do any live editing from the NAS.
Based on my use case, and my understanding of your suggestion - would the following work:
Use rear two drive bays as mirrored boot disk for TrueNAS OS
Pool #1 - Stripped Mirror - 8 Wide VDEV (for XCP-ng VM storage repository)
Pool #2 - RAIDZ1 - 8 Wide VDEV (for general file shares)
Pool #3 - RAIDZ2 - 8 Wide VDEV (for general file shares)
It would be nice to know the workload of the VM’s running on xcp-ng.
If it were me, based on your requirement from what we currently know, I’d set up an:
Pool #1 - 2 wide 4 VDEV mirror for backend storage
Pool #2 - 8 wide 2 VDEV Raidz1
In this way your archive of videos and light share traffic will get a small bump in performance and have the redundancy of 2 disk failure (one in each vdev). Also are able to get a higher capacity than raidz2.
Oooh, I like your recommendation as well. But help me understand your option 1 a little better. Are we foregoing resiliency in favor of performance? What would happen to my VMs if I lost even 1 disk, do the other 3 VDEVs continue?
Workload is pretty light really. We don’t use the building during the week, so we’re there mostly on weekends. Here’s a screenshot of my VMs - nothing that’s doing anything intensive:
In mirrors you are allowed to lose 1 drive per vdev. Which means you can lose 4 drives all at once if there was a single failure in each vdev and you can still be up and running. But if you lose both drives in a single vdev then it would be a bad deal.
But the chances of losing both drives in a single vdev is low in my opinion. As long as you use good enterprise drives. Not sure where you are buying you drives from, but serverpartdeals.com has pretty good prices on used/referb enterprise drives. They even come with warrenties.
The old recommendation was no more than 8 disks per vdev.
Only concrete thing I’ll tell you is mirrored boot drives from the back ports. Probably only need like 256GB drives for this, that will hold a lot of older operating system as updates happen and make it easy to go back if there is a need.
If you put things into 8 drive groupings, I’d probably do all RAIDZ2, I just don’t think you are going to see any benefit from a 4 wide mirrored pair.
And in all honesty, I’d probably build a smaller storage server just for XCP-ng to keep the VMs separate from the user storage. Then you can also back up the VMs to that larger user storage. A 4 drive server will be plenty fast enough for most use cases unless you have big databases, and then those should probably live on the other storage anyway. 6+ drives would be best, but four SSD, especially NVMe or U.2 would be fast enough for a 10gbps connection.
