So, my brother and I live on opposite ends of the Country and we’re trying to have our NAS’ backup to the other’s NAS. I got my TrueNAS mini R already filled, but my brother has 18 2.5” open bays in his used enterprise Dell server. So, we watched Tom’s videos, we tested our potential setup (using Syncthing), and we made our plan: a raidZ2 made up of 3 Vdevs that are 6 discs wide. Because they don’t make reliable 2.5” HDDs, we went with SSDs. Since big tech has already taken the good ones, I was stuck with the leftovers from three different manufacturers. All drives were supposed to be 4TB. However, when my brother tested them, 8 drives reported being 4.1TB, and 10 drives reported as 4TB. As we are now forced to rethink this, I thought I’d reach out here to find out the following:
-
Is it better to have a raid Z2 w/ 3 Vdevs of 6 discs each, w/ ZFS counting them as all being 4TB (the ones w/ the “extra” 0.1TB getting lost)?
-
Or, is it best to have a raid Z2 w/ 2 Vdevs of 5 (4TB) discs each, plus a 3rd Vdev of 8 (4.1TB) discs?
I would appreciate any advice or feedback regarding this. Thank you!
Firstly clarifying the terminology / topology:
A pool is made up of one or more vdevs, and the vdevs are used in a sort of striped fashion (chunks to be written are distributed across the vdevs), and at the pool layer there is no RAIDZ2, or mirror, etc. - it can called similar to a RAID0 but it is not actually a RAID0.
At the vdev level is where you would select RAIDZ2, mirror, etc. and each vdev is independently responsible for keeping all blocks assigned to it in healthy and retrievable state.
So a pool with multiple vdevs that are each using RAIDZ2 is like a RAID60 in traditional RAID topology terms.
For your question, I would go with the first, because when one of those drives die it would be easier to be able to grab any 4TB class SSD versus needing to find one that has the special 4.1TB size. The extra little bit doesn’t matter, and in fact will allow the drives to be ever so slightly better at sustained wrote speeds and longevity by having extra blocks that are never used by the OS.
1 Like
Thank you for your reply @brwainer. Appreciate the clarification. Especially since different people and different websites write out same setups in slightly different ways. That’s what we were thinking as well. Having different Vdev sizes (widths), from what I read, could lead to ZFS utilizing the Vdev’s differently, or filling them unevenly. Either way, I appreciate you taking the time to help me. It’s a lot easaier to address issues now, while the pool is empty, then later, after we start using it.
Mixing those slightly different SSD sizes doesn’t really change much in practice, since ZFS will just align everything to the smallest capacity anyway. What stood out more to me is how uneven vdev widths can impact distribution over time. Keeping them consistent feels way safer for long-term performance and rebuilds. I’d definitely stick with a balanced layout and treat the extra space as a bonus zfs layout planning