I have 2 Xen hypervisors on Linux with virtual machines running off an NFS export from the FreeNAS server. The virtual machines are raw img files stored on a ZFS dataset. Previously, I looked into snapshotting virtual machines with LVM which led me to the below link published directly on the Xen project wiki:
The process outlined in the above link includes; pauses the virtual machine, takes a snapshot of virtual machine, mounts the snapshot, syncs the snapshot to backup media, umounts snapshot, removes snapshot, resumes virtual machine. With ZFS I know I can achieve better functionality with snapshotting/replicating but would like to go about it the correct way following the Xen standards.
I have a couple questions:
- Do Xen virtual machines have to be paused for snapshotting?
- Out of the box, will ZFS replication give me reliable backups of my virtual machines?
2a. If not, are there any hooks for the ZFS replication or in FreeNAS to allow a custom script (I.e. one to pause/resume a virtual machine) to be ran before snapshotting? Or would this process even be needed?
If I remember Tom’s video correctly and you are using XCP-ng no they do not need to be stopped. Search for the video it’s rather recent.
Have you looked at Tom’s video about using Xen Ochestra for backups? That seems like it would be way easier.
Not they don’t need to be paused. Use Xen a good bit saddened day-to-day going VMware
I know in Tom’s video he went over using the backup solutions that XCP-ng/XOA provide (such as continuous replication). Is there another video that I have not seen that goes over a FreeNAS/ZFS component?
What about the replication at the block level that ZFS provides through FreeNAS? Would it be safe just to use ZFS replication to a secondary storage server without the Xen server being aware of the snapshotting/replication?
If you use FreeNAS / ZFS to snapshot without the hypervisor or underlying operating system being aware you could end up with corrupted files in the backups in ZFS due to files not being fully written at the time of the snapshot. The cleanest backups are from a shut down VM which can easily be scripted, our methodology varies based on the systems that need to be backed up. Full OS backups are nice, but we only do those where there are larger changes made to the OS. We produce a lot of data throughout the day so we have all those systems creating hourly encrypted backups of just the data and sending it to a FreeNAS server via SyncThing that then goes to an off site SyncThing for backups.
so not a snapshot but just a regular data backup?
I don’t understand what you are asking.
Im sorry I made it complicated to understand, but I mean that not doing regular snapshots of the machines just doing backups of the data inside of them, and only snapshots when big changes happen correct?
Thank you, this is good information. I definitely can understand how shutdown virtual machines are the cleanest to backup as there aren’t any writes being performed when it’s shutdown.
I would imagine I could look at the code to figure this out but is there a spelled out process on how the XCP-ng project is able to achieve the continuous replication? Do they pause the virtual machine momentarily when taking a snapshot? Does pausing the virtual machine ensure that any writes in the queue are fully written to disk?
You need to have the XenTools installed and they have plenty of documentation on their site for how this works. https://xen-orchestra.com/blog/xen-orchestra-backup-concurrency/
Recovering from a ZFS snapshot isn’t rocket science or anything but certainly annoying to have to mount the snapshot, mount it to your hypervsor as another datastore, move your VM over, and then unmount etc etc… So if there’s a supported option to properly back it up I would go with that. Veeam has 2 free licenses that might support what you want - their NFR license (meant for labs, mostly full featured) or their community edition (full feature with no support).
@mikensan The Veeam bare metal products do indeed work. I’ve been using both the Linux and Windows versions (Full disclosure I am a Veeam Pro Partner) the paid licensed versions add the ability to connect the full Availability Suite. If used in production buy the license $100 for server $50 for a work station. Well worth the price they are storage agnostic backup to a USB drive, network drive, even cd. The feature I like is that it creates a boot media an the backs up the entire system and your data great for disaster recovery. In case of a DR just pixe boot to the boot media created or plug in your USB drive follow the on screen get a coffee. Very useful for laptops.
My BU target is a FreeNAS.