Truenas Special Vdevs question

Computer Hardware & Server Infrastructure Builds
1

First time poster - Long time youtube watcher.

On truenas, I have a Zpool made of 12x12tb HDDs. 2x vdevs 6 wide raidz2.

I have a media dataset which contains a majority of my storage. (1mb record size)

My other dataset (64k record size) is shared with proxmox via NFS and attached to VMs (raw files) as storage for my docker volumes. My VM OS’s run on m.2 storage, but all of the docker volume info is stored on rust. Some of my containers are like Plex, Immich, Gaseous, etc. Some of these databases are pretty big. Multiple people use my Immich server. I like passing this through Proxmox instead of direct attaching to VM just for PBS to capture it instead of zfs replications. It’s been easier for me to restore VMs with multiple drive this way…just a heads up if anyone asks why I don’t just do replications.

Anyway…I’ve been experimenting with special vdevs. I recently bought 2x optane p4800x 750gb drives. I put 1 in as a slog and 1 in as a l2arc. I see a TON of sync data on the slog which is nice. NFS is all sync writes. However, after googling it, it seems the l2arc might be a waste of an optane drive when I have 128gb ddr4 ecc ram. Gemini suggested using the second optane to mirror the slog, but even if it fails and corrupts my docker databases, I can just full restore from backup, so that seems like a waste. (Yes I’m aware that i’ll only ever use 15 out of 750gb on the optane drives as a slog. I don’t have any m.2 slots available on my motherboard and the price i got these p4800x drives for was about the same as I would have paid for a lesser capacity drive). I think I might benefit from a metadata vdev but know I need that to be a mirror, and the slog is paying off for me big time.

So…my question for the experts-

–Truenas 26 is coming out next month and I heard it’s using openzfs 2.4. If I’m reading this correctly, if I were to use both optane drives as a metadata vdev and remove my SLOG, the ZIL would get moved to my optane drives anyway??? So I’d basically have the benefit of a metadata vdev, and a SLOG with just the 2 drives? This would be best case scenario for me right now. I really don’t want to partition the optanes to make both vdevs since it’s not standard in truenas GUI atm, and I do have a handful of old 600gb datacenter SSD’s that I could use for a mirrored metadata vdev but hurts to know i have 700+gb of super fast optane not being used.

If any of you can clarify that for me that would be amazing.

Post I’m referencing. Hope I’m allowed to post other sites. Not trying to get banned on my first post.

https://www.reddit.com/r/zfs/comments/1mxnf3y/zfs240rc1_released/

image
image1159×372 41.9 KB

2

I took a look at the ZFS pull request 17505 Allow and prefer special vdevs as ZIL by amotin · Pull Request #17505 · openzfs/zfs · GitHub

As I understand is this introduces the logic for indirect sync writes. To visualize how this changes the “flow” of data in your pool:

  1. The “Old” Way: A synchronous write (like a database update) comes in. ZFS looks for an SLOG. If it doesn’t find one, it writes to the slow HDDs. Even if you have a Special VDEV for metadata, the intent log (the safety copy) ignored it.
  2. The #17505 Way: ZFS now says: “I don’t have an SLOG, but I do have a Special VDEV. Since that’s fast flash storage, I’ll put the ZIL there instead of the slow HDDs.”

Assuming this makes it into TrueNAS 26 this would be a good use for a pair of those drives.

2 Likes
3

How do I go about getting the metadata from my existing datasets onto the special vdev? I don’t know if a zfs-rewrite would work. I know for the 3tb proxmox dataset I could just do a move virtual disks on proxmox but how about my 38TB of media. I don’t think I have 38TB free to just replicate. Do I need to babysit it and move portions of the dataset back and forth? Is this going to make Plex have to rescan all of my data?

4

I have a video on it but the TL;DW is add the metadata VDEVs and use ZFS rewrite.

5

Yeah I watched that one twice and the ‘Metadata-Special Vdev’ one once. I blame Gemini for giving me doubts lol. I know I need to remove my snapshots before I start. Just didn’t want to start a 38TB rewrite and beat up my drives without being certain it would work.

Thanks Tom!