I’ve done a lot of reading/testing, but I can’t figure out which would be the best fit for a small business with 4 hosts.
Inventory:
2 - dell 2950 with 8 logicals core total CPUs 32GB of RAM and 6 x spinning disks SATA 2TB
1 - dell R710 with 24 logicals core total CPUs 144GB RAM and 6 x spining disks SATA 2TB
1 - dell R820 with 80 logicals core total CPUs 512GB RAM and 8 x 1.6TB SSD (new one not installed yet)
Currently we are using ESXI on 3 hosts. running pfsense + some VMs etc… the main issue is that we don’t have HA available and no common storage point. I would like change that to have something more scalable.
I have checked to migrate all our hosts on XCP-NG instead of ESXI. Since XCP-NG is free to use, and have a lot more feature than ESXI, its look like to way to go.
I would like to use a file system that can create a big pool with all those storage HDD/SSD. In December, XCP-NG have publish a beta about XOSTORE. I know that is based on linstor and drdb, but it’s look like XCP-NG want’s to have it supported directly inside their tools.
We create VMs for customer from time to time and for a certain period of time. After the contract is finished, we delete the VM and that’s it. I have checked openstack/cloudstack over XCP-NG and this is looking very interesting also.
What I’m looking is to use 2 pfsense in HA with all 4 hosts in XCP-NG pool. The part that i’m not sure is the storage. What kind of storage can I do to have a huge pool and assign some GB to each VM ? This need also to have some sort of replication.
I have not tested XOStore yet, we use TrueNAS as our storage system with XCP-NG. You can run an HA setup with pfsense but I still prefer to use pfsense on bare metal hardware rather than virtualized.
I think I have used pfsense for more than 8 years now on a VM, and I don’t see any problem with that setup for now.
For the TrueNAS, do you run Truenas as VM with PCI passthru on every hosts ? How can we setup a storage pool with a minimum of HA (some replica) with TrueNAS, do you have any video/blog on that subject ?
I Start using XOstore, For now, It’s look promising!, I’m not sure for now that HA is working for VM in a pool if the VM its hosted on the XOStore. It don’t seem to work.
Are you looking for “hyperconverged” solutions like vsan? Is that level of availability critical enough at the cost of storage to you? Depending on your storage profile configuration the amount of available disk space could equal the smallest disk pool on one host.
Distributed storage offers high availability at the cost of complexity, when it breaks you will definitely want support - it is a nightmare. This is speaking from experience with Dell’s vxrail solution. If you don’t get support then hyperconvered / distributed storage isn’t the right choice. The time spent during downtime to fix it will null any “potential” benefit.
That said, an alternative would just be silo storage on a truenas box that gets sync’d to another truenas box. Disaster recovery would be just pointing hosts / services to the second truenas box. This can also be automated via clustering in TrueNAS Scale or failover in TrueNAS Core.
Also not to be “that” guy but a modern i3 desktop will put those 2950’s to shame despite the logical core count difference. The energy cost to run those for a year would afford a new desktop + SAS/SATA controller.
@mikensan Thanx for your input,
Not so encouraging!!!. I’m looking for hyperconverged. Since we don’t pay for electricity, this is not really an issue for the old 2950!.
Very lucky on not paying electricity costs, I’m jealous!
There are several strategies to distributed storage using hyperconverged hosts, but to have a VM be redundant it needs to exist on at least 2 hosts - which further reduces your available storage.
So if you have 3 hosts with 10 TB each and you want a fault tolerance of 1, you really only have about 10-15TB available depending on how you setup the policy.
Depending on your needs this may be the solution, just good to be aware before making a decision and buying more stuff.
The 2950s are definitely underpowered, it’s going to be a big ask for them to both support VMs and manage the disks (parity calculations / checksum / indexing). Albeit, some hyperconverged solutions do support adding volumes from raid cards - a lot want to manage the disks directly via passthrough / HBA. If you’re not getting the speeds you expect, likely those 2006-2008 era processors are to blame.
I’m think using the raid controller on each server. I see no advantage of doing all of disk parity/computation in software.
On those 2950 I have a PERC6/i raid controller, On the R710 I have an H710 Raid controller and for the new server (not arrived yet), I don’t know what is the raid controller.
The redundancy of 1 VM is enough for me. I don’t mind to not run VM on the 2950s and only use them as storage in the hyperconverged thing.
Probably all our VM will be running on the R710 and R820 server.
You know, the only whish I have is to be able to add storage if needed too. If all my storage are full I want the possibility to add “easily” an new storage and expand the storage.
We don’t have the 100K$+ budget to do the right thing. I’m not in a position where my VMs need up time of 99.9%, I can live with restarting the VM manually on another host is something go wrong. All those VM are private, no other than us needs it.
I’m testing the XOSTORE (based on linstor) really hard in lab. Up to now it’s look promising. The integration with XCP-NG are not yet finished (in beta stage), but from my testing, it’s seem to work. XOSTOR hyperconvergence preview | XCP-ng and XO forum (some of the issue raise by me “dumarjo”) are there too.
I also looked at TrueNAS Scale and from my understanding, this is in beta stage too. It’s look like this is a real need, but not easy to do !
Well if there aren’t plans to run VMs on the 2950s, can’t hurt to flip them as stand alone storage servers instead of hyperconverged.
The PERC controllers are tanks, but make sure you have a spare one laying around so you can import the raid if need be. They’re crazy cheap at least. Otherwise if you run them as standalone storage servers then having software defined raid (RaidZ / mdadm) won’t be too bad especially if you only need gigabit throughput. My xcp-ng is running off a single gigabit card with a docker host running roughly 20 containers.
The advantage of using software defined storage is your ability to yank out the hard drives and stick them into a new server. Using a controller means the controller must go with them, as normally you can’t import the disks into other controllers. You can make upgrades with RaidZ pretty easily so long as you keep your vdevs reasonable. Groups of 3 or 5 means you only need to buy disks in batches of 3 or 5.
It’s good you’re testing, but make sure you test failure as well as functionality. Create scenarios that may happen (unplug the power on 1 host / unplug ethernet on another / etc…) and see how recovering from those disasters are.
ZFS is very much becoming mainstream to where xcp-ng and proxmox both offer support one way or another. It’s come a long way over the past 6-7 years that I’ve used it.
To that note, take a look at proxmox as well. It integrates very well with docker, zfs, and supports clustering.
What’s the main advantage you’re looking for with distributed storage (VSAN / XOSTOR / Gluster)?
I’ll will look into the RaidZ. I have not read anything about this. I’ll need to look at it.
I know that software raid is more compatible than the hardware raid. Again, RaidZ is intriguing !
This is the main objective in the testing. Crash recovery.
I will.
Well, from myI understanding, it’s the possibility to have a single storage where everything can be put in that cluster. The other thing is the possibility to have 2 copy of each file/block over the cluster. So if one host go down, everything continue to work in the time to get back the host online.
Is this true ? Do I miss something about the (VSAN / XOSTOR / Gluster / ceph) ?
That’s good that is what you’re testing, so many people just look at features and functionality and never consider recovery.
For your objective “put everything into that cluster” - the main purpose normally isn’t to have the largest pool possible, it’s redundancy. So for example don’t expect 3 hosts with 100TB to give you 300TB when configured for failover.
While creating one large data pool (striped data across multiple hosts) is a possibility it requires a pretty serious backbone to ensure all the storage hosts can put the data together. I believe the folks over at storinator have some videos on youtube of them doing this via glusterfs / ceph. This is actually what backblaze does in the end. They add entire hosts to expand their storage, it’s pretty cool.
That said nothing is stopping you from creating such a pool and just seeing if it will fit your needs. Honestly would be a cool project to do!
This will come down to storage policy and how you configure the fault tolerance to find a balance of what you need.
And yup correct - the idea is if a host just dies that the VM will either continue running (hypervisor HA + Storage HA) or that it will simply start the VM on another host (Storage HA without hypervisor HA).