I am trying to play around with my truenas settings to better optimize different datasets based on the data actually within them - this is not a huge deal, but why not try and maximize where possible.
I was looking at my NVR data (from Frigate) and I am shocked to see LZ4 currently has 1.23x (so 23%?) compression ration on this data which I did not expect, possibly because some of it is snapshots not just video files? Based on this, would it make sense to change from LZ4 to ZSTD or ZSTD-Fast?
I record NVR data to my main (and only) truenas box and its only pool, but this is just a homelab environment with mostly me as the only user, this Frigate data is 98% of the time the only data even hitting the machine. TrueNAS is virualized with 8 Xeon 2660 v4 cores and 60 GB of RAM - plenty for what I ask of it. I also do have a SAS SSD for SLOG and a second for L2Arc not that that has much bearing on this dataset. Sync writes are dissabled for the dataset and this is a NFS mount on the NVR system.
Are you shocked that the compression (1.23%) isn’t higher, or more compressed? Or that the data is being compressed that much, or at all? Not sure what your cameras are doing with the data they’re sending out, but media generally doesn’t compress well unless it’s in its raw form.
A simple test of how LZ4, ZSTD[x] or ZSTD-Fast[x] compare would be doing a simple replication or rsync to datasets with the different compression types and levels you’re curious about. Note the compression you get from each one. Don’t forget to reference the CPU load both during the compression phase, but also the decompression phase when you’re consuming said media (at least if keeping CPU usage down is important).
These video files are already compressed, getting 23% compression on them is actually very odd, especially if they are h.264 or h.265 files. Even mpeg2 generally doesn’t get compressed from standard zip or similar compression, because it already has a lot applied to it. Unless things are being re-encoded, then all bets are off.
The fact TrueNAS is reporting 1.23x is odd, and I honestly don’t entirely understand why or how… maybe for whatever reason frigate has a lot of compressible data besides the video streams. I’ll have to hunt that down on the frigate end I assume. But the fact it is compressing so well, does it make sense to try a different compression mechanism to reduce the amount of data being written to disc?