The following will expose my ignorance concerning xcp-ng and FreeNAS.
My understanding is that iscsi is logically equivalent to a shared scsi bus with multiple initiators, and that it provides block level access to a device.
I don’t understand how it would be possible to have multiple host with r/w access to the same device without a cluster aware filesystem. The defunct Digital Equipment Corporation developed a distributed lock manager for VAX/VMS Clusters, which allowed coordinated access to shared devices. Here’s a link to a high level overview of VMS Cluster communications from one of the VMS developers, Andy Goldstein. Specifically what was done to make sure that cluster membership was always consistent so that uncoordinated access to the directly attached storage was prevented, as he describes here.
Does xcp-ng have a cluster aware file system? If not, how can multiple systems safely access the same device with read/write access using iscsi?
I am not looking for a tutorial, that is well beyond the scope of a forum, but if there pointers to where I could find this info. I have used google, and from what I found, it seems to say that you can’t have multiple host with read/write access to iscci storage without using a cluster aware file system. Wikipedia has this Clustered file system does not have anything I saw about xcp-ng but does mention VMware VMFS and RedHat GFS2. Windows clusters take the easy way out, and limit access to a “shared” volume to one host at a time.
If the XCP-NG hosts are in the same pool, they can share NFS or iSCSI storage but they can no both write to the same VM at the same time. Because in this configuration they share a storage pool, when using Xenmotion to transfer a running machine only the RAM of the running machine needs to be moved to the other host and once completed on host stop writing to that VM and the other starts.
I cover this in my XCP-NG HA Video, hope that helps.
So with all this in mind, what would the best way be to move vms from one host to another?
Background: We’ve got an older server that we’ve been using for xcp-ng for the last year or so maybe that is connected to freenas with iscsi for our vms. We now have some newer hardware setup with xcp that we want to make the main host. I’ve gone through and setup another extent and associated target in freenas and started to connect the new host but is this really what I want to do, that would essentially make a whole other “disk” on in freenas, right?
End goal being that we will use the newer hardware primarily with iscsi and then set up the older with it’s own drives and setup backups. What would be the best direction for this? HA wouldn’t really work right since we only have the two servers. Would added them to the same pool give access to the same iSCSI location and then as long as the vm is shutdown, it could be moved to the new host?
If I am understanding your goal correctly, put the two servers in the same pool and replicate the data over from the new drives to the others as backup.
Yeah, that’s about it. I guess where I’m getting hung up is connecting the secondary machine to the same iscsi target on freenas, since we are keeping the vms stored on freenas, in their current locations. Should I just have to create another extent and associated target with the same mount location? (If that makes sense at all, haha)