Truenas Core - replacing all drives with smaller capacity?

I’ve been searching and most posts say this can’t be done, but most are also a few years old.

I have eight 10tb drives that I’m going to want to cycle out in a few years, we are not using them to capacity (approx 3tb out of 46tb). I’d like to go with cheaper 6tb drives or maybe even 4tb drives.

Is it possible to offline and replace the drives 1 by 1 to end up resilvered back at a smaller capacity? These are user shares so permissions would get all out of whack to copy/backup and delete the array and start over smaller.

I don’t think TrueNAS (or any ZFS installation, for that matter) will let you replace drives with lower capacity ones. But I haven’t verified or tested this, it’s just my intuition.

What I came to comment was actually that if you do go the route of creating the new pool from scratch, I see 2 possibilities how you can avoid the issue with the lost permissions:

  1. Making use of ZFS, you can replicate datasets from the old pool to the new pool, provided you can have all the disks online at the same time (if not, you could also replicate over the network). As long as a dataset fits on the destination pool, ZFS will happily replicate it from a larger pool to a smaller pool. This would be my preferred solution.

  2. Many file transfer tools (like rsync) have archive modes which will preserve permissions, timestamps, etc. giving you essentially a 1:1 exact copy of the filesystem.

I didn’t think about replication, I might be able to do this to another server.

I’m finishing replacing old drives in one of my servers and may decide to try the downsizing in my lab, only one VM running in my lab right now so I can burn it down. Lab is mixed 500gb drives, old drives from production are 1tb drives. So I can resilver up to 1tb drives, then try going back down to 500gb drives with little penalty save for wasted time if it doesn’t work. That sounds like a test for next week. Also important is that my lab storage is up to date with Core, so better chances of working.

I had a chance to try offline and replace with a smaller disk… It does indeed throw an error that it can’t replace with a smaller disk.