Is there a way to completely delete all past xcp-ng backups and start over on shared storage? Trying to clear what appears to be a massive coalesce issue and I am fine with starting totally over. Running fully patched 8.3 to a fully patched Synology nfs share. I tried moving the backup data to another folder inside the share and deleting all backup jobs but the coalesce list is still showing and no new backups with run due to VDI chain protection. Do I have to remove the shared storage config in XO?
Unfortunately I’ve tried that and still run into the same VDI chain protection error.
From the file system level you can go into your remote storage target and delete all the backups, but this sounds like you may have too many snapshots attached to VM’s of which you can delete those as well. If you stop the VM’s and let the system idle it will coalesce on the storage.
1 Like
After more thought and testing, I’m coming to the conclusion that when the VDIs are stored on a NAS, backups either to the same NAS or a different NAS always run into coalescence issues that never resolve.
When I run the VDIs off local xcp-ng host storage and back up to NAS, coalesce is not an issue. I can temporarily fix coalesce by migrating the VDI from NAS (NFS SR) to local storage and back which runs the coalesce as part of the move, but the issue always recurs when the servers are running on a NAS NFS share.
Is xcp-ng known to not run well for backups when servers are running on a Synology NFS share and backed up to another NAS?
I am running XO from sources and the network is 1Gbps. I try to spread backups out to avoid too much happing at the same time. My hosts are all running 4G of RAM and when I look in logs I don’t find exceptions or memory issues when looking for my coalesce is not keeping up. My VMs run 24/7 so I don’t want to shut them off to try and help the coalesce work better.
I can also “force” the backup job to run using the provided button in the backup log summary when I get the VDI chain error, and it works, so if this is the case it would be great of xcp-ng had an option to always force backups when the VDI chain issue exists. If that is a feature I haven’t seen it.
The coalescence occurs based on how busy the SR is, perhaps post in their forums if there is a threshold that can be set.
Just a thought I had while reading the thread: it seems that you are storing the backups on the same storage device as the VDIs for your VMs. While this might provide recovery when some VM gets shredded, it is not resilient agains failure of the storage device. You then might (at least for a while) lose access to both, the VDIs AND the backups. If you have more storage devices you could consider using different storage devices for the VDIs than for the backups.
Absolutely, and that is precisely what I’ve done. For example, I have a Win11 machine VHD living on Synology with its backup going to TrueNAS running on a different host. So far so good after a couple of days, but if I recall correctly, even this scenario wasn’t working for me on 8.2.x and I was “forced” to store all disks on local xcp-ng host disks and backup to a NAS. Previously that was the only way I could coalesce and not run into constant errors on job due to inability to coalesce. I recently updated my xcp-ng cluster to 8.3 and XOCE to the latest build. That is the only change to my environment really. I seem to also recall Vates did a lot of work on improving backup code in 8.3. If I can get this working without modding my hardware stack I will be a very happy homelabber!
1 Like
Sure enough, this morning I see the unhealthy VDI chain for this backup so it is off the rails again. I will have to move the VM disk to local storage on one of the hosts and backup to one of the NAS. Frustrating, and I have no idea how to make this work.