FreeNAS ZFS Pools RAIDZ RAIDZ2 RAIDZ3 Capacity, Integrity, and Performance

When setting up ZFS pools performance, capacity and data integrity all have to be balanced based on your needs and budget. It’s not an easy decision to make so I wanted to post some references here to help you make a more informed decision.

                       capacity
                          /\
                         /  \
                        /    \
                       /      \ 
          performance /________\ integrity

These are the metrics to consider when deciding how to layout your drives in FreeNAS.

  • Read I/O operations per second (IOPS)
  • Write IOPS
  • Streaming read speed
  • Streaming write speed
  • Storage space efficiency (usable capacity after parity versus total raw capacity)
  • Fault tolerance (maximum number of drives that can fail before data loss)

Do you need more storage? more speed? more fault tolerance? How you lay them out will cause dramatic differences in in these numbers which is why deciding this is the first step in your plan.

In ZFS, drives are logically grouped together into one or more vdevs. Each vdev can combine physical drives in a number of different RAIDZ configurations. If you have multiple vdevs, the pool data is striped across all the vdevs.

Here are the basics for calculating RAIDZ performance, the terms parity disks and data disks refer to the parity level (1 for Z1, 2 for Z2, and 3 for Z3; we’ll call the parity level p ) and vdev width (the number of disks in the vdev, which we’ll call N ) minus p . The effective storage space in a RAIDZ vdev is equal to the capacity of a single disk times the number of data disks in the vdev. If you’re using mismatched disk sizes, it’s the size of the smallest disk times the number of data disks. Fault tolerance per vdev is equal to the parity level of that vdev.


TL;DR: Choose a RAID-Z type based on your IOPS needs and the amount of space you are willing to devote to parity information. If you need more IOPS, use fewer disks per stripe. If you need more usable space, use more disks per stripe.

For those of you that want to dive deeper into this topic, here are some great write ups that really helped me understand this much better.

Six Metrics for Measuring ZFS Pool Performance Part 1

Six Metrics for Measuring ZFS Pool Performance Part 2

ZFS STORAGE POOL LAYOUT PDF

The ZFS ZIL and SLOG Demystified

ZFS RAIDZ stripe width, or: How I Learned to Stop Worrying and Love RAIDZ

ZFS Raidz Performance, Capacity and Integrity
https://calomel.org/zfs_raid_speed_capacity.html

2 Likes

Hi thanks for your video(s), love them!

Hopfully you can advice me on something VDEV related :slight_smile:

I run a small film production studio and currently we’ve been using a couple Synology NAS systems (5, 8 and 12 bay units all running RAID 5 + backups). We currently have roughly about 244TB’s (x2) of video material.

The main function of the Synology’s at the moment is file sharing / backups which work completely fine. But I want to switch to something more powerful like unRaid / FreeNAS for these reasons:

  • Expansion options
  • More power —> CPU and graphics pass tough (that’s why I think unRaid is a good option)
  • Virtualization / renders system (thinking about Ryzen or even Epic processor + beefy GPU)
  • Faster network speeds, (I currently have 10gbe, which for now is good)
  • more spinning disks / more data / more bandwidth (right?!)

My question; let’s say I invest in a 24 bay (like a net app DS4224). What is the best file system if you want to gradually expand your data 1 or 2 disks at a time, given the fact I can’t afford to buy 24*14TB (2x including backup!) drives at this point.

I would love to take a closer look at FreeNAS or unRaid + ZFS (with 1 or 2 disk redundancy) but I understand you can’t put more disks in a VDEV that already exists, so in order to expand your capacity you always need minimal 3 disks (right?!)…

Hope this makes sense :slight_smile:

I prefer ZFS but you are correct about the issue of just wanting to expand as needed. I don’t use unRaid but as I understand it does offer good support for passthrough and virtualization. I also think unRaid will let you expand as need https://wiki.unraid.net/Add_One_or_More_New_Data_Drives so that might be what works best for you.

Thanks for the quick reply!

  • Let’s say I want to add another VDEV (in Z1 or Z2) you need minimal 3-4 disks right (like RAID5 / 6)
  • In the triangle (you sort of said this in your video also) should be one factor added: ‘costs’ :slight_smile:

Basically data is an expensive ‘hobby’ haha…

Cost is affected by all three factors. I would saw draw a circle around the triangle and the closer you get to any of the points on the triangle, the higher the cost. :slight_smile:

1 Like