XCP-NG VS Proxmox 2022

Computer Hardware & Server Infrastructure Builds
4

I watched… thanks for that. I’ve tried XCP-NG a couple of times and couldn’t get things working the way I like quickly enough to get me to move from proxmox which I converted to back when ESXi took a dump on me.

Anyhow, I wanted to note that backups in proxmox don’t need their backup appliance at all. I added a NAS network share as an available storage source (NFS, SAMBA, iSCSI, etc) and so long as you designate it as being available for VZDump Backups you can setup scheduled backups and snapshots into your already existing resources from the proxmox web management client or the cli.

1 Like
5

Here is an excellent post by @olivier.lambert who runs Vates, the company behind XCP-NG & Xen Orchestra covering some very important details regarding the history and architecture of the hypervisors

6

Haha. I see we spammed the XCP-NG video comments about proxmox enough to warrant this video. I haven’t watched this video yet but thanks Tom for covering this. I actually am running both XCP-NG and proxmox in my home-lab but haven’t hammered on it enough yet to really make a decision on what would work better, and which has the potential to scale into enterprise environments better.

7

Really nice comparitive analysis. I’m a hardened Debian user of 20 years, and have never ventured to CentoOS or RedHat. But XCP-NG to me offers a construct that will bring me over to CentOS for the first time.

I see through the analyis that XCP-NG offers a far more powerful solution, in terms of how you can intricately design, develop, and continually evolve how your system management operations work. Whereas with Proxmox I can see a clear glass ceiling.

I’m always of the philosofy do it right, do it once. So I will certainly invest up front in XCP-NG as a platform with far grater runway, and much deeper potential.

1 Like
8

While I concur with your comment, I will not say that one is absolutelt superior to the other. Both XCP-NG and Proxmox have a place in enterprises of all sizes. For the SMB market space my preference is Proxmox since it is a much simpler install, has a good interface for admin and has better support for ZFS, LXC containers.

Business owners are questioning VM Ware’s commitment to the smaller markets that have been using its products for years. To me this is a signal to market alternatives.

1 Like
9

@g-aitc — How’s your experience been with LXC containers? I’ve never used them. Usually just defer to Docker if needing a containerized application.

10

So far so good. Client has a very well versed in house IT manager who has put together a great team of contractors and consultants. they were buying a line of bus app that was LXC wanted some guidance. no issues.

1 Like
11

I’m finally getting around to actually testing the differences (and similarities) between xcp-ng 8.2.1 and Proxmox VE 7.3-3.

Hardware specs:
HP Z420 workstation
Intel E5-2690 (v1, 8-core, 16-thread, HTT enabled, 2.9 GHz stock, 3.3 GHz all core boost, 3.6 GHz max boost)
8x 16 GB DDR3-1600 ECC Reg RAM (I think it’s either Crucial or Samsung memory?)
Samsung 850 EVO 1 TB SATA 6 Gbps SSD (it will work sufficient for some basic testing)
GeForce 660 in one and GeForce 980 in the other (I don’t think that it really makes a difference because I’m not passing the GPUs through to the VMs)

The Test:
So far, I’ve installed SLES12 SP1 and trying to just set up a basic Apache web server.

Proxmox has already finished the OS install and the system is up and running. Total time is probably around maybe 15 minutes or so. Somewhere around there.

xcp-ng on the other hand, I still haven’t gotten SLES12 SP1 installed yet. The default SLES12 SP3 template seems to make the graphical console really, really slow, so I’m not really sure why it is so slow and what’s the hold up.

But yeah, so far, Proxmox at least already has the system up and running. shrug

edit
I’m installing SLES12 SP4 now instead…

edit #2
It would appear that Proxmox, on my HP Z420, is consistently faster, at least as far as boot times goes, than xcp-ng at an average of 36.156 seconds vs. 52.324 seconds after 5 trials.

Interrresting.

Going to be probably doing more testing with it maybe with some other OSes, but that’s SLES12 SP4 with apache2 enabled and running and just serving the default index.html page saying that apache2 works.

12

Single VM performance is not the best best workload type to measure as most of the time people are going to use a hypervisor for many VM’s. There is a follow up discussion this topic here Xen vs XenServer vs KVM vs Proxmox

13

The gap is indeed more visible on old and very vulnerable Intel CPUs (Xen is doing a lot of mitigations to be secure as possible, unlike Linux/KVM and “open bar” driver model like virtio).

But on modern CPUs, like EPYCs, the story is a bit different :slight_smile:

Also, XCP-ng storage stack is a bit more sensitive to the clock speed and efficiency for various reasons (mostly the defence in depth layers for the PV driver models)

1 Like
14

I know.

My Windows system is currently hosting I think 10 VMs. Something like that.

I gotta start somewhere.

(And even then, I had to re-install the SLES12 SP4 VM in xcp-ng a few times because the system would appear to be unresponsive at times, but because by default, it formatted the primary drive using btrfs, so when I force it to shutdown (due to the unresponsiveness), it would then boot back up, but whenever I reboot it, it would have to reboot twice.)

SLES12 SP1 (because it doesn’t have a template for that) failed to install in xcp-ng completely.

Most of my VMs (especially my Windows clients) are running Windows 7, and there doesn’t appear to be a template available for that.

(So I’m not sure that if you are doing say some consulting work, how you would test Windows 7 clients in xcp-ng without the template being available for that.)

I found it “interesting” that it also appears that if you want to create a new VM without a template, that it doesn’t really appear to give you a great option to do that. Or at least, it’s not as simple and straightforward as I assumed it should be (because if you have the local SR defined, and you have the network defined), I don’t really understand why then, when I create a new VM, that I can’t leave the template field blank, and then fill in everything else to create a VM where a template does not exist (e.g. a Windows 7 VM)?

Yeah, I am testing with the hardware that I have available to me and because I want to try and make the cross-comparison as fair as possible, therefore; I am testing it on two identical systems (systems that used to be compute nodes in my micro HPC cluster).

The test of deploying SLES12 SP4 as a apache server is because it’s just one of my use cases where I have a dumb web server that hosts files for my clients to download data from me. It’s easier (for the clients, who might not be technologically inclined) than using FTP for file/data transfer.

The other thing that I am also unsure of is, is there a way to “detach” the console so that it would be in its own separate window so that I would be able to monitor the performance of the VM, but not having to have one web browser window open for the console and the other browser window open to monitor the “general” “tab” of the client VM?

(I’m coming from an Oracle VirtualBox on a Windows host perspective/point-of-view and I am testing both proxmox and xcp-ng simultaneously. I haven’t even gotten to the point where I am trying to import Oracle VirtualBox .ova files in yet. I’m just testing some more “basic” functionality like “create a Windows XP VM”. And “create a Window 7 VM”. etc.)

15

Few notes:

  1. Issues in the guest aren’t related to the hypervisor (like btrfs by default and such, are choices of the guest). Never heard of problems with SLES, but I can’t tell for all hardware/software combo :smiley:
  2. Templates are “basic” things with a configuration (“kind” of BIOS settings if you like) that are a bit different between Windows and Linux for example. You can use “Other Install media” when you don’t know what to use. But mostly any recent Linux template is fine and very close to each other (so eg an Ubuntu or Debian 11 template will do fine too).
  3. Detach the console will be here for the next UI (XO 6)
16

Agreed (and disagree at the same time).

Yes, btrfs is the choice of the guest/client.

But the VM/guest freezing up though (or seemingly or apparently looking like that it froze) which necessitated the forced shutdown and then powering on the VM again – that, I would think, would fall under the purview of the hypervisor.

(And a part of this is because I am doing like a direct compare-and-contrast between proxmox and xcp-ng simultaneously, so whatever I am doing on one hypervisor, I am doing the same thing on the other one, with maybe about 1.5 seconds lag (as fast as I can move the mouse over to click the same buttons, at approxmiately the same time.)

Proxmox didn’t exhibit this freezing issue.

xcp-ng did so, at least twice so far, whilst trying to configure/get the HTTP server running, and going through multiple reboots so that I can time them to see which one would be faster. (It was a way for me to test the performance, albeit, not a GREAT way to test it, without using ANY other tools/software than just the OS/guest itself.)

I know, but I can only report on my observations, as they are happening.

(I was only enabling and disabling the HTTP server in SLES12 SP4 via yast (the GUI) and when that become unresponsive, I would try it via the terminal inside the guest and it took would become unresponsive, and then I would go to “advanced → force shutdown” and after the guest/VM powers down, I would immediately start it back up again.)

(and for the HTTP server configuration, I would just uncheck the box to listen on port 80 on the loopback interface, and then click next, next, next, until the end, and then click on the option to start the apache2 server on boot, and then close/finish that).

And then the only other thing that I would do as well would be to delete the network bridge (as it only has one NIC interface anyways), and then set the static IPv4 address, subnet mask, gateway, dns servers, and hostname of the guest/VM. (Did this in both xcp-ng and proxmox.)

That’s all that I’m doing with it. (And then to test the reboot times, or actually to test whether the HTTP server would start on boot like it is supposed to, I would disable the HTTP server, and reboot, and then time the reboot, and then go back into yast to see if the HTTP server started back up like it was supposed to (or not), and if it didn’t, I would enable/start the HTTP server service and reboot it again, and then time the reboot again as well.

Rinse. Repeat (to collect repeatability data).

xcp-ng has a standard deviation of 5.660984 seconds whilst proxmox has a standard deviation of 1.09739 seconds (on the reboot times).

The max/mins (in seconds) are as follows:
xcp-ng: 61.75/45.96
proxmox: 37.22/34.08

Initially, I would have thought so as well.

But seeing how xcp-ng appears to have failed to install SLES12 SP1 using the SLES12 SP3 template, my observations would suggest or indicate that I might not actually be able to do that. If I was able to install SLES12 SP1 using the SLES12 SP3 template, I would observe the data that would abide by that.

However, I ended up switching to SLES12 SP4 because I wasn’t able to get SLES12 SP1 installed in xcp-ng. (In Proxmox, I was able to install SLES12 SP1 no problems.) I waited about an hour (maybe close to an-hour-and-a-half) whilst the installer for SLES12 SP1 tried to boot up in xcp-ng before finally calling it as that it failed to execute the SLES12 SP1 installation, and that’s when I stopped the SLES12 SP1 VM in proxmox, deleted it, and then re-created new VMs on both that are both SLES12 SP4 where in xcp-ng, I was using the SLES12 SP4 template.

Being new to this, I have no idea why this was happening, but due in part to this, I ended up testing the system until around 4:30 AM this morning.

:slight_smile:

But I would surmise then, that until that is released, I would need to use multiple browser windows to be able to control the system and monitor the VM’s/guest’s performance?

Would that be correct?

(And this is with the assumption that I am not installing any additional software in the guest/client VM (e.g. TigerVNC or TurboVNC or something like that – this is just using what’s available to me, as the user, out of the box.)

(Sidebar: proxmox already allows you to detach the console using noVNC that’s built into proxmox)

17
  1. It’s impossible to tell why it froze with the level of details :person_shrugging:
  2. About the your benchmark: it’s very vague and again, depends on a lot of factors (storage, CPU, spectre/meltdown mitigations, even power settings in your BIOS could make a huge diff). You have different results because by definition Proxmox and XCP-ng are vastly different (Xen vs KVM, storage stack, isolation and security policies, scaling, etc.)
  3. No problem to use multiple tabs. We also use noVNC. You can also use XO Lite for very basic management like you are doing.
18

Being new to it, and looking at it from myself as an end-user perspective, the “test” was either “pass/fail” or a “no/no-go gauge”. (i.e. either it works, or it doesn’t.)

Also being new to it, I didn’t try to spend any time, debugging it. As a test VM, creating and then removing VMs as necessary (to try and get it work) was significantly faster at 4 AM than trying to do any sort of debugging/investigations.

I can try and see if I can reproduce it to see if I can get the system to generate the debugging information for you, but I would need a LOT of guidance to be able to do that, not knowing how it all works, under the hood/behind the scenes.

I agree. It is. But it’s something that I am able to test “out-of-the-box” without installing any additional software.

re: differences
Well, this was why I specifically chose to test this on my two HP Z420 workstations because they have about as identical of hardware as you can get unless you get into the very tiniest of minutae (down to silicon lottery). As for the power settings, because both compute nodes were in my micro HPC cluster, therefore; both were tuned for performance rather than power savings, for example.

Yes, I understand that there are fundamental differences between proxmox and xcp-ng, but that is also the basis and the premise of the test.

(i.e. one might be more secure, but at the expense of speed vs. the other, where it might be less secure, but also comes with the benefit of the increased speed)

I’m not a security expert, so I wouldn’t know how to test the two hypervisors on that basis. But for performance, that IS something that I am able to test, noting that some of the reasons for the differences in the apparent performance might be due to things like one is more secure than the other, as well as all of the other differences that you’ve mentioned as well.

(If I knew more about security, then I might be able to perform a better compare-and-contrast analysis between the two hypervisors, but unfortunately, I don’t.)

For my Windows 7 VMs, I interact with them a LOT, through the GUI. I surmise that for like, Linux server tasks, XO Lite might work (since a lot of linux sysadmins interact with the system via the terminal/ssh anyways).

On the other hand, it is VERY difficult to shop online, via only using the Lynx text-based browser (that you can use via said CLI). It’s not impossible, but it would be very cumbersome to do so. (Again, this is because of the Oracle VirtualBox perspective where I am coming from.)

19

Are you a home or small business user? If yes, Proxmox is probably the better package. On a single server or in small clusters, it comes with a lot more “out of the box” goodies that small busineses or home users typically need. You can for example use it as a basic VDI-host relatively easy. You can build clusters and replication relatively easy, you can tweak and script many things relatively easy, etc…

If you are a larger business, XCP-NG with Xen Orchestra is probably better suited and more flexible and scalable, and maybe also more performant on appropriate hardware, depending on your use case. (Allthough Linux/KVM is getting better and better :slight_smile: ) However, if you use it at home or in a small office, you will probably miss some of the typical home user / small business features, because larger organisations typically use dedicated solutions for these use cases anyways, like for example their Virtual Desktop Infrastructure.

1 Like
20

I’m probably more of the home user than a large business (although, I CAN envision a scenario where as a large business, for example, all of your front line workers (e.g. bank tellers)) just needs something small and lightweight to be able to connect to the xcp-ng hosted guests/clients do go about their jobs.

I mean, I can’t tell you how many times I’ve walked into a local branch of a bank, noticed that they’re still using Dell SFF desktops, and think to myself “you know, you can virtualise that”.

Like, they can deploy a system that’s running xcp-ng locally in the branch, and then all of the tellers can just connect up to that rather than using one of those Dell SFF computers.

Or, alternatively, if power and internet were more reliable, they can have said xcp-ng running in their regional offices or maybe even at their corporate head office/data center, and then you can have one server hosting hundreds to thousands of VMs, and since the bank teller software isn’t very computationally demanding nor resource intensive, you can massively oversubscribe the server hardware and you’d STILL come out on top.

(Or like use an 80-core ARM CPU on the host xcp-ng server.)

So many possibilities.

re: Proxmox for home
I think that I was reading (or watching a YouTube video somewhere) which stated that unfortunately, Proxmox can’t import .ova files from Oracle VirtualBox.

So, either I am going to have to re-create all of the VMs from scratch, manually, or there’s a way to like extract the VDI file from the .ova file, and then somehow convert that so that Proxmox would be able to read and use it.

I haven’t gotten that far yet.

21

Sorry – more dumb questions from my side:

The host system of the xcp-ng, with my 1 TB Samsung 850 EVO SATA SSD, has something like over 800 GiB free.

I read in the documentation that if I create a local ISO SR, that it would only be like 15 GB or so because it would use the space that’s available on dom0.

Is there then, a way to create a NFS share on the xcp-ng host itself (which would physically reside on the same Samsung 1 TB SSD), and then let dom0 use that as the shared ISO SR?

If so, how would I go about doing that? Is that something that I can do through XOA or would I have to do that in the command line of the xcp-ng host?

(i.e. create the directory, edit /etc/exports, add the entry in there, and then enable NFS service on the host?)

22

Aight, if that is your main use case, probably neither Proxmox nor XEN Orchestra will make you happy. Or in other words, neither the accompanying standard management tools like Xen-Orchestra or the built-in management interface of Proxmox will probably have enough features to provide such an infrastructure for your end-users to use it as their main workplace, at least not without some additional tweaks. Whether it is better to use XCP_NG / XEN or Proxmox / KVM in the backend would again be a separate discussion. But I’m not an expert when it comes to VDI infrastructure, so I don’t have any product recommendations.

You definitely can import OVA disks to Proxmox, but afaik you have to create the VM manually.

1 Like
23

I dunno.

I’m just testing both of them out right now.

Like I said, I’m coming from Oracle VirtualBox, so I’m just trying to see what are the pros and cons of each; what I can do, what I can’t do, etc.

I’m neither buy nor selling on any particular solution, so long as in the end, I am able to do what I need to or want to get done, regardless of which hypervisor (or whatever you want to call Oracle VB) I use.

I’m agnostic to the solutions. The one that will be able to help me accomplish what I am currently doing with Oracle VirtualBox will be the solution that I’ll end up deploying. So, if xcp-ng can do it, then I’ll use that. If Proxmox can do it, then I’ll use that.

If they can both do it, then it’s a bit of a toss-up/wash.

If neither can do it, then I stay with Oracle VB.

Hope this makes sense.

Thank you for your help, input, and insights.

1 Like