XCP-ng-video was really usefull fo me and i as home user have tried all of those from hyper-v through ubuntu kvm,proxmox… XCP-ng really nailded it for me.Anyone else out there who want to share your experiences about these different hypervisors?
1 Like
I tried using windows hyper-v and It was good for windows based VDI but other than that, I found it wasn’t good. The windows OS uses alot of system processos and really isnt as efficient as I was looking for
XCP-NG is great. I tried it in my home lab for a while and works great for what it does. The only issue that I found with it is its lack of support for exporting to different hypervisors. Using OVF, I was unable to import to ESXi. Even just trying to connect the virtual disk to another hypervisor didn’t work right. It was a while since I tried this but it took me a bunch of wrangling to get it to work in another hypervisor (This was done as an exercise to see if it could be done.)
This observation is probably pointless, since most use cases don’t involve exporting to a different system, but its good to know none-the-less.
The 2TB limit on their virtual disks also kinda bothers me. I wish they supported VMDK. But maybe one day! =)
I really like XCP-ng as well. One thing I had issues with is support for a Linux based server manager. In Windows 10, XCP-ng Center worked great, but I haven’t been able to find a stable alternative on Linux. I tried OpenXenManager, and it somewhat works, but frequently crashes, contains outdated dependencies that were a pain to get, and lacks the full functionality of XCP-ng Center. I have Xen Orchestra set up and that’s what I use for now, but it also has some non-functional features. For example snapshots don’t work correctly, even though it worked fine on XCP-ng Center.
So what are some functional Linux alternatives?
That’s a good observation regarding the exporting to different hypervisors, for home lab I think its more common for us to want to move VMs from one system to another since home labs are subject to way more change and tear downs.
I’ve been so far deep in VMware since the mid 2000s that I haven’t played around with too much. For more complex setups there is licensing that is needed but that hasn’t been an issue for the deployments I’ve had to do. I’ve looked at the other setups and they seem well but I just really love VMware. VMware just seems polished in my opinion while the other solutions have left me lacking features or certain management utilities.
1 Like
I’ve used VMware in production deployments and I have had mixed success with it. I have faced issues with windows server 2012 hanging up on reboot on some VMware ESXi.
Its a rather robust platform don’t get me wrong. I’ve seen some awesome stuff built with VMware. 
The one thing that turned me off from them was the vCenter. It can be a pain in the neck loading it up for the first time. But its such a necessity, especially if you wanted to manage a number of physical hypervisors with one central UI. I’ve even had vCenter go as far as causing some corruption of my deployed “datacenter”.
I am using an evaluation copy of Windows Server 2016 with Hyper-V but I am going to switch to XCP-ng. Whenever I login to the Windows server it says that Updates are available…
I have faced the same thing with server 2016. Have you setup a patching time for it? I know that that can be of some help with the log in notifications. Do you connect to the hyper-v instance remotely via hyper-v manager as well or only via RDP?
Been using Citrix XenServer for the last 8 years at work since version 5.5. Done some virtual gpu allocation with NVIDIA Grid and Tesla cards. Really enjoy the Xen platform, and our use case doesn’t warrant the need to use ESXi, and I don’t want anything to do with Hyper V 
@ctech It’s a lab environment without much configeration done. I guess I could setup autoupdate for some time at night but I’d rather switch to a software that has not to be updated as often. I mostly access the server directly or via AnyDesk.
I am currently using Vmware ESXi, because my school provides free licenses for me to use it. I have to say i have not experienced that many issues with it. How-ever the web-ui is buggy at times and a refresh fixes this, v-center is nice, but also takes quite a bit of RAM (and it get’s tight when running ms exchange). Personally XCP-ng on the other end is something i am not really a fan of. Mainly because of storage. If you want to add another local storage device (last time i checked) this has to be done through the CLI. don’t get me wrong, i love terminals. It just seems weird that there is no option built-in to the windows application. Also there is this minimum amount of storage required for VM’s with specific OS’s like freebsd. otherwise besides that, i think it’s a great hypvervisor. And look i am not the smartest kid out there & there are probably ez solutions to all of the points i just made, but hey that’s how you learn something everyday ;).
This bothered me also. I must admit, I had to set up a win10 in virtualbox to manage xcp-ng. I did compile the community edition of xen orchestra, tom posted a video on how to get that up. You can do most things in xen orchestra but it does have bugs, particularly with changing networking. It’s probably fine for minor changes in a production environment but for home lab where machines and networks are being moved, created, lots of network interface changes? XOA community just was laggy, buggy, and the UI is not the right fit for me. I too tried openxenmanager. That has not been developed in quite some time and only functioned partly. The only machine that runs windows is the one virtualbox to talk to xcp-ng. Windows doesn’t like that I won’t let it talk to WAN. If I can find some time I will post bug reports to the XOA folks cause it’ has been getting better. GL @drowsy
That also bugged the hell out of me when I tested it. I tend to like to store ISOs on the hypervisor as opposed to sharing it out from a NAS or Samba share. That’s probably like I am more comfortable with ESXi
VMware is indeed polished and a defacto standard. It’s biggest problem is its cost especially if you want the bells and whistles. Most small businesses will run the most basic edition. VMware is a cash cow for Dell and they will milk it until somethig better at a better price point shows up on the radar. XCP-ng looks promising and works but needs more but I do like what I see.
I started using XCP-ng thanks to Tom’s videos. Before that I was using the Bhyve stuff on FreeNAS. It worked. Mostly. But it was hard to do some things. They had an issue where it wouldn’t run Ubuntu 18.x versions of Linux. The issue had been fixed in the upstream FreeBSD for a while, but not in FreeNAS.
So I bought a refurbished Dell PowerEdge R710 and a couple of Mellanox Connectx-2 10G network cards and set it up to use space on my FreeNAS through iSCSI just like Tom recommended in his videos.
Copying the VMs over to XCP-ng from Bhyve was… challenging. I wound up using a bootable image of Clonezilla in each VM (the old Bhyve one and the new “emtpy” one on XCP-ng) and just let it copy the contents over just as if they were two physical machines. Worked great once I ironed out a few details.
Curious as to the details you had to iron out.
@g-aitc: I had to decide what type of save and restore to do. I finally decided to do a system-to-system copy (can’t remember their exact terminology). One VM is the “remote” (send) and one is “local” (receive). I first set up a fresh Ubuntu install on the XCP-ng VM, just so it would set up the disks, install the bootloader, etc. Figured that was easier than trying to remember how to do it all manually. (Also, I saved a copy of the /boot/grub/i386-pc directory contents away for use later – more on this below.) Then I specified that I wanted to do a “partition-to-partition” clone, not a whole disk clone. I set the sending side up first, then it waits for the other side to connect. It helpfully prints the IP address so you can specify it in the receiving side (it lets you set it up networking via DHCP or static).
Now to get Clonezilla to boot up in the Bhyve VM, I first had to add a CD-ROM image. It took some doing to figure out how to actually get it to boot from the CD, since it always wanted to do the virtual hard-drive first. I finally set the VNC connection to “wait on boot” (not sure why that would make a difference on boot order). After several reboots if finally gave me the Clonezilla GRUB prompt off the CD image. If anyone knows how to get it to let you pick the boot device, I’d love to know. I Googled around but never found anything.
For some reason, I had to boot up the “Failsafe Mode” on the Bhyve side. I couldn’t get it to boot all the way if I didn’t. The default mode worked great on the XCP-ng side.
After doing the copy, the new clone didn’t boot. Under Bhyve it was using the /boot/grub/x86_64-efi directory, but was using /boot/grub/i386-pc on the XCP-ng side. So, as I mentioned above, after installing the fresh Ubuntu, I saved the i386-pc directory away in a tar file on another machine. After doing the clone, I used one of the “Exit to shell” options in Clonezilla. I manually mounted the cloned virtual hard-drive, copied down my saved tar file, and unpacked it in /boot/grub.
Next, while in the shell, I edited the /etc/fstab file (mounted as /mnt/etc/fstab). In my Bhyve VM, I was mounting the root filesystem by UUID (by using the /dev/disk/by-uuid/ entry). Since the UUID was different on the cloned version, I had to find the correct one in /dev/disk/by-uuid and change it in /etc/fstab. While I was in there I modified the swap location as well. Even if you’re not using UUID to mount the partitions, you still have to edit the devices specified in /etc/fstab because on the Bhyve VM (for me) they showed up as /dev/vda and on XCP-ng VM they showed up as /dev/xvda.
I think that was all I had to do. Let me know if you try it and have problems. Maybe I can help.
At home I use proxmox, before I used debian with kvm (proxmox almost the same, but with gui and a whole lot more).
at work we use vmware (vSphere appliance) and I control a customer with XenServer (not xcp-ng) .
I might even have a chance to introduce proxmox to future customers, cause not everyone wants a costly OS for their VMs and I will not recommend Xen (in any way) cause Xen is a bit weird and not as straight forward as Proxmox (my experience) also, I do not like applications on a computer or a VM/appliance for VM management (vsphere/xen orchestra)
At work we had VMware, ESX, but it was an older version and we don’t have a large demand for VM use beyond some network appliances. Using Tom’s videos, migrated to Xen Server, and then to XCP-NG. Along the way learned about vLANs, back ups, and a whole lot more that Tom helped educate us on. Much healthier network and backups, as well as unlocking all 32 CPUS and 256 GB of RAM on our Dell R810 that was previously capped due to our ESX licensing. Much happier these days, and our conversion was slow, but the benefits far outweighed. We did use CloneZilla to migrate most of the VMs.