Apologies if this is not the right forum for this question but I am curious as to what alternatives folks with day jobs in IT are exploring now that Broadcom has resorted to extortion pricing for VMware? My company has a farm of about 300 ESX servers and 3000+ VMs connected to a FC SAN with Pure and Netapp arrays. I’m not sure what we’re actually looking at for VMware alternatives but I probably need to start researching since I’m a storage guy. At home I play with Proxmox and Truenas but I really have no idea how scalable they are in a corporate environment. For those whose employers are thinking of jumping ship and leaving VMware, what are you looking at to replace it? Thanks!
I’d take a look at XCP-NG, Lawrence Systems has solid content on it. I run it in my homelab and am working toward deploying it in production. It scales better than most people expect, but you’ll want to test storage integration, HA/live migration, and support options. Really requires testing and what is required from the org.
https://www.youtube.com/results?search_query=lawerence+systems+XCP-NG+VMware
2 Likes
I can’t speak for the scalability of Proxmox but we have done many larger deployments of XCP-ng. They do offer corporate support and SLA options.
2 Likes
We’ve got 26 hosts under our xen instance. Migrated our production cluster over from VMware last year.
While I have been using xen + xcp-ng for years it’s a new system for my co-workers, but they’ve started to like it far more than VMware.
1 Like
You really need to look at all of the different VMware products and options that are deployed. XCP-ng is easy to work with and might be a good choice if it fits your use case. Once you tally up all the VMware products, it might be worth contacting Vates to discuss if it is possible.
Assuming you are also a Microsoft account holder, what size license do you have and how many of those duties could be handled with hyper-v at no additional cost?
Do you have a big container workflow with Tanzu? That might point you towards Harvester HCI and Rancher.
At the price where I work, $120 per core, 300 servers at minimum of 16 cores is going to be a lot of money. VCF 9 will make the licensing a little more flexible as it seems to remove minimums from the payment scheme, but still core based. You are licensing total core count, split across hardware as you see fit.
I would guess that the majority of places looking to migrate were using VMware Enterprise + without really any of the additional options. If this is the case, then moving to XCP-ng and or Hyper-V should be a “simple” migration.
In theory, Windows will handle container workflows too, but I’ve never tried even simple Docker containers with it yet. But Tanzu was a different beast being Kubernetes management. Again theory is that Kubernetes can run on XCP-ng, but never tried it. I think I’d learn Harvester or Rancher based before I tried to run Kubernetes on XCP-ng, Rancher (and Harvester on top of it) was built to handle these workflows, not applied on top later. Running Rancher/Harvester along side XCP-ng certainly seems workable, but also kind of a duplication since you can run VMs on Harvester. The hyper converged part might be an issue, I haven’t looked into Harvester enough to know if it supports iSCSI, NFS, SMB, and FC storage. I would like to think it does, but I think Nutanix is still building this out from their HCI only ways of the past.
Nutanix and Hyper-V seem like the popular options for converged infrastructure. You could also look at RedHat OpenShift, Openstack, or Scale Computing HC3.
Thanks for the comments and experiences. For background I’m a SAN/Storage admin and all of our ESX hosts are currently FC connected to Brocade and Cisco SAN switches. I would guess that over 90-95% of the guests are Windows. Lets just say that our Windows team is vehemently opposed to iSCSI for whatever their reasons are (that pre-date me) so any non VMware solution would be FC. Has anyone done a migration from VMware to one of the aforementioned hypervisors in an FC SAN environment? I know VMware on FC SAN is solid but we don’t have any other hypervisors connecting to the SAN today. Thanks again for your time.
My migrations involving NetApp or Pure were usually moving to NFS or iSCSI. I still maintained separate switches for the storage traffic. When I worked with EMC or 3PAR it was on FC.
I’ve migrated between ESXi and Hyper-V numerous times and deployed VMware, RedHat, and Hyperflex in greenfields.
I used the Cisco/NetApp Flexpod validated design guides for much of the architecture.
BTW, fun name 
Redhat Openshift? Have we forgotten what they did to CentOS? Unless I were running on Redhat, I wouldn’t touch it with a 10 foot pole. I still stay away from Oracle due to the Open Solaris thing over a decade ago.
Hyper-V only because a lot of corporations already pay for unused licenses. Most A/E 3 or A/E 5 contracts will have extra for a cluster.
If I refused to touch a companies products because they did something I didn’t agree with in the past, I’d likely only be using brand new ones until they do the same.
For those of you who have already migrated away from VMware how has your experience been with your new vendor? Are they scalable to large VM farms (if 3,000 VMs is considered large? maybe medium?)?
FWIW there is a legacy lack of confidence in our network so any storage over IP is not being given serious looks by our Virtualization Team (NFS, iscsi, NVMe over IP). There was an announcement by Netapp recently regarding their partnership with Nutanix that said they were looking at FC protocol with Nutanix by late ‘27 so that might be a light at the end of the tunnel for us.
If their is a lack of confidence in the network, and you are using vmware, are you using vSan? if so, how do you have it connected?
I friend of mine is working on a long term project to move from vmware to Harvester/Rancher/Kubernetes… It’s going but there is some roughness, they use a SAN for everything and Harvester being HCI doesn’t use a SAN for anything but backups. He said SUSE has some scripts to make it work, but it’s a mess. I liken this to the problem with XCP-ng and migrating larger than 2TB VMs off of VMware and onto XCP-ng, split the data from the OS and let the data live on shared storage and the OS on the HCI storage.
Otherwise, they are really happy with SUSE support and have been using SLES for years for their Linux flavor of choice. And all that said, Harvester seems very young, while it gets work done, they are growing very fast and Nutanix might be a more stable platform right now if you want something different from XCP-ng or Proxmox Datacenter. You could spin your own with KVM (etc.) or go to Xen Project and DIY a system.
If it were me, I’d concentrate on the networking and get that to be solid. Enterprise 25gbps is not that much money (as a minimum these days), but a lot of things might work “OK” with just 10gbps depending on the design. HCI storage really should be as fast as you can afford and separate from VM traffic (even then don’t expect blazing fast until nvme over IP or other advanced storage protocols). I’m afraid I don’t know much about fiber channel or FCoE, something about direct connection so it looks just like a local disk and generally high speed, etc. but that’s where my experience ends, I’m on the low end of things where I don’t have the money to really play with stuff like this.
Thanks for the reply. We’re a reasonably decent sized “enterprise”. For at least my dozen+ years here we have always been FC SAN for all Windows and AIX servers, and the same when we implemented VMware many years ago. Without airing too much more dirty laundry our Network, Sysadmin, and Storage teams are pretty silo’d. Nobody knows what anybody is up to until they turn it on and find out something doesn’t work and they should have asked the other guys for input.
Silos aside, quite a few years ago we did acquire a Nutanix system when we picked up a new site but since the Virtualization team did not want to get into the storage business and the storage team did not want to manage hyperconverged storage, it got mothballed. Nowadays they are more budget conscious and we have implemented VMware VSAN at remote sites where it made sense to not make a big storage/san purchase.
We are now 95% VMware ESX with several hundred ESX hosts, a handful of Windows servers, and a dozen or so AIX boxes all connected to 64G Brocade SAN Directors, Netapp and Pure flash and spinning disk arrays. We do have several PB of NAS on Dell/EMC Isilon and the Netapps, too … probably about 12PB of storage in total.
To the best of my knowledge we’re not running anything exotic (though we might be doing so in “the fog”) so our VMs are application, web, and SQL database so I think our virtualization needs are pretty generic, but reasonably sizeable.
To go back to your network comments, we do have multiple 100g pipes between sites and for the most part are at least 25G within sites, if not 100g, but the Virtualization team is only asking for block LUNs for their testing for potential replacements for VMware.
Apparently it’s hard to break an old FC SAN habit and test with IP based storage, but the virtualization vendors are pretty focused on it. I know that Pure had an announcement with Nutanix to support NVME/oIP and it will probably eventually support FC, too, like Netapp so with any luck we can hold out until that happens. Ironically I run Proxmox at home connected to Truenas serving NFS datastores … but what do I know?
I keep an XCP-ng + Truenas system set up in my lab for “emergency” work I might want to do.
I think you are somewhat of a worse situation to my friend with the Harvester testing, they are pretty heavy with the SAN storage for everything, never went vSan because it probably cost way too much per TB or PB of storage and you only get a third of that payout (assuming 3 copies).
And that said, I thought Nutanix now has NFS and SMB storage for VMs, they were talking about that a couple years ago when I first started looking. They are on my list to learn a bit, but Harvester jumped in front because it just felt more compelling. Then I found out my friend is actively developing the Rancher/Kubernetes end of the equation, a coworker is handling the bigger Harvester and cluster build. My friend is prototyping this in a lab I gave him a while ago, glad that old stuff is still being useful to him.
My career has evolved from a mainframe tape/card jockey, to mainframe sysadmin, to open systems, to storage. My home lab was a windows pc and has now grown to 2 Proxmox servers with a dozen LXC and VMs (mostly Ubuntu), 2 Truenas servers, 10G network w/DAC cabling, and a pfsense firewall (soon to be Opnsense), Some old dogs can learn new tricks. Some enterprise dogs have a really hard time with that.
I think you might like OPNsense, I’ve had the business license for 2 years now.
I just set up a VM with Opnsense and have been watching any videos I can find. Ultimately I’ll move it to a small Dell desktop with 2 NICs. I’m reasonably sure its basically “the same” as pfSense just a different cover. My needs are simple for my homelab: router, dhcp server, firewall rules to keep all the Chinese cameras from phoning home every 5 minutes, separate VLANs for my stuff and the neighbor on my wifi, rules for blocking inbound nasties, haproxy, wireguard. I *think I’ll need AdGuard to mimic what pfBlockerNG does unless there is another option. Also not sure if it has an equivalent radius server add-on or built-in yet. Still learning.
It’s not really the same as pf, hasn’t been for a long time.
There is dns blocking in the free version now, but you can do another ad blocking plugin too. You could also try Zenarmor free plugin which catches a lot of stuff, so some choices for blocking. There are a lot of good guides on the zenarmor site, and the forum is great too. I like it better than pf when I was running it.
Apologies if I rubbed you the wrong way on that comment about them being the same. I am hoping that they do “similar” things. 