At home, I am operating a NAS running TrueNAS SCALE. I have 2 pools:
tank
- 4 x 4 TB WD Red Plus (WD40EFRX, these are the CMR ones)
- Raidz-1
- 10.44 TiB usable capacity
- currently about 65% full
- used for storing media, backup and archival data
ssdtank
- 3 x 1TB WD Red SSD (WDS100T1R0A)
- Raidz-1
- 1.75 TiB usable capacity
- currently about 15% full
- used for VM disks, container volumes and personal files
The machine has 32 GB of ECC RAM. My network is currently only running at 1 Gbit/s and that’s definitely a bottleneck for some applications (mostly, copying files). I plan to upgrade that in the near future.
Bot for now I am looking for advice on a different matter: I have a couple of extra, used, consumer-grade 120 GB SSDs lying about and plenty of available SATA ports. I was wondering if I can put them to use sensibly on my NAS in special VDEVs, i.e. metadata, log or cache. I am specifically looking to improve performance, not to extend capacity.
These are my thoughts:
- Basically, it makes no sense to add any special VDEVs to the SSD pool since it’s already flash storage. It is not going to get any faster.
- I could add a cache VDEV to the HDD pool. However, the files on that pool are mostly not accessed repeatedly (i.e. during video and audio playback) except for maybe when going back and forth through photos. And also media files don’t necessarily need to be read faster than HDD read speed as they are usually played back in real time. Currently, there are no issues playing back media and skipping to random points in a film both via VLC over SMB and via Plex is basically instantaneous.
- I could add a metadata VDEV to the HDD pool. Since failure of the metadata device means failure of the entire pool, I would use a 3-way mirror of 120 GB SSDs. But I have no idea what kind of performance gain I can expect of this. When navigating through the media library using Windows Explorer over SMB, there is often a noticeable (although, not deal-breaking) delay before the contents of a folder are displayed (say, 1s up to 5s in extreme cases). Is this an area where a flash-based metadata device could help?
- Finally, I could add a log VDEV to the HDD pool. I am thinking 2-way mirror of 120 GB. This will only improve synchronous writes, which by default SMB is not anyway. Asynchronous writes will go to the RAM as long as there is any available. But I can force synchronous writes via the dataset settings. I have actually tested this: Copying a 10 GB file from an SSD on my PC to the HDD pool with forced synchronous writes yields 30 - 40 MB/s. On the SSD pool, I get 120 MB/s (limited by line speed). So I can see a SLOG be an advantage in these scenarios:
- when I copy data that cannot fit in RAM
- in the extremely unlikely event of a sudden power-off (only helpful when combined with forced synchronous writes)
So, this has been a lot of text. I would be very grateful for your opinions and comments on the matter. Another question I have is about removing any of these potentially added VDEVs down the line. I know for a fact you can remove cache and I cannot see any reason why I shouldn’t be able to remove log (although I am not certain). But I am unsure about the ability to remove a metadata VDEV.
