[First post! Hope this is the right category. This also posted on truenas forum. But Ive been watching Tom’s videos for about a year and thought I’d get feedback from you guys too.]
OK. So I am currently using a single truenas scale server for both storage and VMs.
TO BE:
Two servers: truenas server for storage and xcp-ng server for vms
Truenas and xcp-ng servers connected via 10g ethernet (perhaps via crossover cable, no switch).
Truenas (with more storage) for xcp-ng VM storage and backup.
Xen Orchestra running in VM on truenas.
Then setup truenas backup to some sort of cloud storage (backblaze, etc) which I dont have now, but probly should.
My current truenas is on a 2U, 8x 3.5" bay supermicro server with 4x3tb drives in raidz2 (1 pool, 1vdev, boots off internal SSD). I want to replace these drives with say 8x6tb drives in raidz2 (for example) and use the old 3tb drives on the xcp-ng server (probably something like a dell r720 with 8x 3.5" bays).
I’m a little unclear on the process. I will have the spare “to be” xcp-ng server to help with the transition before xcp-ng is installed on it.
Thinking of a migration process something like this:
backup existing server config. backup the data somehow if possible.
install the 8x 6tb drives in the spare server and temporarily install truenas scale and set them up in 1 raidz2 pool with all 8 drives in 1 vdev (dealing with disabling RAID via JBOD etc on new server).
shut down all the VMs on the existing truenas server.
use zfs replication to move the data from the existing server to the new pool in the spare server.
export the new 8-drive pool in the spare server.
export the old 4-drive pool in the existing server and remove the drives.
move the 8 drives from the spare server to the existing server and import the pool
once it is comfirmed that all is well, use the old 4 drives for xcp-ng in the new “spare” server. If something isn’t right, reinstall the original drives and import the original pool.
I’m not sure if this is The Right Way to do it. If not, please let me know a better way. Also any input on the direction I’m heading is appreciated.
Some questions I have about this method:
My main pool is not encrypted, but most of the data (including VM zvols) is in an encrypted dataset that I manually unlock after a boot. Anything special I need to do with the encrypted dataset during this process? I’m guessing that it will just replicate with encryption intact. I don’t need unlock it on the spare server. It just needs to unlock when the new pool is imported back into the original server.
Will all my existing truenas VMs still work properly? I would guess so, but I just dont know if they would become “disconnected” from their zvols somehow when the original pool is exported and replaced.
Will all the snapshots be preserved in this process? I would guess so.
Will all the network configuration, users, permissions be preserved? I would think so since that’s in the server config on the boot drive and not in the zfs pool, right?
After the zfs data migration is complete and xcp-ng is installed on the new server, I should be able to use something like clonezilla to migrate the VMs from truenas to xcp-ng, right? Is there a better way?
-Take new server and set up a temporary NAS with an NFS share and connect this share to XCP as a VM SR.
-Go into XO and migrate the storage for each VM from old SR to temp SR.
-disconnect from old SR
-build new truenas *alternate put old drives in XCP server and set up storage for local SR and migrate from temp SR to local SR.
-connect VM SR to new truenas and migrate VMs to the new VM SR
Then you can set up your other server for second XCP or whatever else you want to do. Don’t forget that you can have multiple XO connected to your pools, make one a VM on XCP, one a VirtualBox on a laptop, etc.
As far as the local storage with XCP, I haven’t played with it much, something I want to set up because it makes a good place to migrate your VMs when you need to do an update on your NAS. Eventually I’ll deal with that on my production system, but for now I was able to build a second small and slow Truenas on an old server so I can bounce my VMs back and forth depending on who needs updates.
Thanks. Just edited original post for clarity. Not using xcp-ng now – all vms currently running in truenas. Don’t even have new server yet. Workin on it. I just have the one truenas currently and once I get the new one, I will only have two servers total, existing one for truenas, and new one for xcp-ng. Wanting entire existing truenas zfs pool to be on a different set of drives first. Then I’ll setup xcp-ng, VM SR, etc.
Oh… Not sure about this. If you have XCP running, you might be able to boot each VM into Clonezilla, then boot the XCP VM into Clonezilla and do a network transfer to move them. Theory=yes, but I have not done this.
If you can export the Truenas VM as a VHD or a couple other formats, then you should be able to import them directly into XCP through XO. If they are Windows VMs, you might try disk2vhd which is meant for physical to virtual, but I bet it would work virtual to virtual too.
Yeah. Tom did a video on moving VMs between hypervisor platforms with Clonezilla. Basically uninstall the “vmware tools” like stuff and clonezilla it. Install the right vm tools after its done.
https://youtu.be/wSTk9BLwF5k
Got the xcp-ng server ordered tho! Should be in next week along with associated doodads (including an open-box deal I found on an Optane 900p for an SLOG).
Dell R720
256G RAM
2x E5-2690 8 cores, 16 threads @ 3.0GHZ (but I’m upgrading it to E5-2690V2 10 cores)
8x 3.5" bays with caddies
Has 6x 6TB 7200RPM SATA drives in it
10g and 1g NICs
$275 + $80 shipping.
These are the drives I’m gonna move to the TrueNAS server (Supermicro 2x X5680 6 cores@ 3.3 MHz, 192G ram, 8x 3.5" bays).
The pair should be a good foundation for whatever I want to do homelab-wise. Just gotta figure out how to replace the drives in the TrueNAS.
The X5680 is may be a limiting factor with the most recent windows OS, after an update a year ago, my Windows Server VMs started to have reliability issues. I replaced with servers running Xeon E5-xxxx V2 and everything is back to stable.
The trouble started after the February or March 2023 updates, they had been running fine for more than a year before that. I’m fairly sure that the issue was the older X5650 processors that I was running XCP-NG on at the time. I gave these servers to a friend and he is running Linux on them perfectly fine, so not a hardware issue other than lack of support for very old processors.
Thx. Good to know. Aint gonna run no windows server VMs in my homelab unless I absolutely have to or I’m temporarily testing something. The X5680’s are just on the truenas server tho. The xcp-ng server will have E5-2690V2’s.