I’m turning to this forum regarding a Truenas Core build, as I know there is a lot of knowledge on the topic here.
Hardware base:
Supermicro A2SDi-H-TF (as far as I know this is the same board as in TrueNAS Mini XL+)
4 * Seagate Exos X16 ST16000NM001G Hard Drive - 16TB
16GB ECC memory
I’m thinking of running RAIDZ2 which would give me about 32TB of usable.
The motherboard has 2 10GBE network ports, so this would need some cache to be able to fill the bandwidth, and this is where I would appreciate your input. What comes to cache read cache should be fine with SSD’s, write would preferably be Optane, but this motherboard does not support it? I’m looking at Crucial mx500 250 GB drives, as they are reasonably fast/priced I could add a few of them, instead of going with bigger/more expensive, and I would still be limited with the SATA interface.
Read cache L2ARC
This is more likely the easy one. It is my understanding that it doesn’t have to be in RAID mode, however in order to achieve speed from SATA a RAIDZ1 with 3-4 disks?
Write Cache SLOG
I see 2 options:
2 SSD’s in RAIDZ1 (I’m aware of the 100TB write endurance)
Or is it possible/have anyone tried Optane?
Motherboard has one M.2 PCIE Gen 32 and one PCIE Gen 34 slot. Could set them up as RAIDZ1.
Currently I have a dedicated RAID card in my PC, with spinning disks, which this Nas will “replace” so I can also access the files from other computers. I also want to get rid of the RAID card, as I could still get a bit back on EBAY for it.
I have also recently upgraded the machine to a Threadripper Pro running Proxmox and Windows virtualized. Motherboard is Pro WS WRX80E-SAGE SE WIFI, which has 10GBE, so paired with the motherboard of my NAS they should have a 10GB link, that I would like to utilize.
Another thing to note is that as there are long times I know I won’t be working on my machines, I will shut down the TrueNAS frequently.
This will be used for mainly photo editing and by 1 client/user at the time. Basically local drive for the machine.
Tasks:
Ingest photos from memory cards/external drives. File size (at the moment) 25MB. Memory cards go a bit over a 100MB/s read these days with big files. Drives will far exceed that.
When I open the folder with the images in editing for the first time, it starts generating thumbnails, so it reads the files (25MB) and then writes thumbnails(~66KB)/work-preview(~2MB) and a base settings file (~7KB)
When editing, opening folder with the images the next time it reads the generated thumbnails and settings file.
Processing after editing is done, reads the files (25MB) and a settings file (~7KB) processes and writes output file ~50-200MB
I usually ingest 10000images/folder, so the data size there is about 200-300GB, and I will work of that set for a while (not jumping between sets, so should stay in the read cache I presume).
As this is my main PC, I will use it for web browsing, email, cad etc. also. But if I get performance in the photo-editing side, I think it would also reflect on the other things I do.
Backup:
At the moment the backup (another TueNAS) is behind a 1GBE network connection so it takes a bit of time, but this will be addressed when I update my home network to all 10GBE. And I will use my gained knowledge from this to upgrade that NAS later.
I’m also planning on at some point setting up a offsite backup in a garage, however only connection to that will be a 5G mobile modem.
I know there are many ways to do backup, but I’m not happy with a backup unless I know that the write has been verified.
I have used TotalCommander to sync directories, and after that I have done a compare of the contents of the files in the directories. That way I know that the data has been written correctly and is accessible. Have used this method for years, and so far so good. It is quite heavy on the network though.
I see that you have used Syncthing and set it up with TrueNAS. I think I might be able to use it.
However for peace of mind it raises a few questions on how it works:
As Syncthing is running in a jail, I assume you can start and stop it? In other words you can run it when you want and not have it running all the time? Would be important as the backup server will only be on when I do backups.
It is my understanding that TrueNAS uses checksums to verify data? But if it writes data and checksum, it only catches the issue on read?
I would prefer to have a verification of written data. Would a scrub after the Syncthing has done a backup accomplish the same level of certainty?
And if it does, would it be possible to set up a task to sync everything, and then run a scrub on the receiving end? Or would that have to be done by hand?
This would be great as the data would only be transferred over the network once, especially in the case of offsite backup.
General ZFS question:
A question that I have not seen a good explanation for, how it works. I have read that the L2ARC does not have to be mirrored, that you instead can use a stripe, so what happens if it fails?
I mean while it reads the file it is also sending it?
Or is it so that it reads chunks of the files and has a checksums along the way so if it doesn’t add up it doesn’t send the “bad chunk” but instead gets it from the HDD’s ?
My reasoning behind my initial taught about the build:
Because these files are quit large my thinking was to try to make sure that the drive side could “fill” up the 10GBE connection.
That’s why I was thinking of putting 2* Crucial mx500 series drives (which are quite inexpensive) in RAIDZ1 as L2ARC. And If I can stripe them, I would be able to have a big read cache.
As the write cache needs to be able to handle lots of writes, Optane seems to fit the bill. And you can pick up 64GB sticks for $70/stick.
Now I’m not sure if this motherboard could use 2 of them, one in the PCIE slot and one in the M.2 slot ?
However Crucial MX500 500GB is at $35, so for the same money you could get 4 of these (RAIDZ1) , and even if the write endurance is limited it would still require the whole capacity of the NAS to be rewritten several times. And as most of the data is in the original photos and the processed ones wont change or be re-written this should not be an issue after all?
Syncthing can be turned on / off / rate limited as needed
ZFS verifies integrity of each write because ZFS is a Cow
ZFS Scubs also validate the integerity
The cache order in ZFS is to check ARC and then L2ARC, and if the data is not in either spot then it goes to the VDEV to get it. Read caches are only effective for frequent request of the same files.
