Seeking guidance for HA Cluster in business scenario

Computer Hardware & Server Infrastructure Builds
1

Hey all, I’m looking for some help and guidance on a low-end high-availability cluster for a small business.

Currently we have just a single proxmox node running on a Dell R730XD for when we started out in development and production, hosted at our office.

But we are looking for more redundant low-end high-availability cluster to run production environment and place them in a datacenter.

What exactly is a best practice or guideline to follow when it comes to a setup like this?
I was kinda thinking of just getting 3x Dell R730XD or R740XD and UniFi networking and call it a job done?
What hardware would you chose in this case?

Let me know if I should provide more details for our use case and any suggestions for hardware or thoughts for setup is very much appreciated.

Best regards,
Skorbis

2

My first question to many clients is Do you actually need automated High Availability (HA)?

For a small business moving production into a datacenter, true automated HA is often a trap. It forces you into complex storage layers like Ceph, kills local disk performance, and introduces “fencing” risks where a minor network issue could cause your cluster to reboot a healthy server. You can get more performant HA by using a NAS but that adds more expense and a redundant NAS would be even more expensive.

A much more practical, cost-effective, and highly performant route is resiliency via ZFS Replication. I have a video on that here:

Why Manual Failover is Better for Most Small Businesses

  1. Raw Local Performance: By using standard ZFS pools on local SSD/NVMe drives inside your Dell R740s, your databases and apps run at full local hardware speeds because no network lag is holding back every single disk write.

  2. Low RPO/RTO: You set Proxmox to replicate your VMs to the other nodes every 1 to 5 minutes. If Node 1 physically dies, Node 2 already has a 99% up-to-date copy of that VM sitting on its disks.

  3. Human in the Loop: Instead of letting automated code guess why a server went offline, a human admin logs in, reviews the failure, switches the VM config to Node 2, and hits “Start.” Your downtime is maybe 3 minutes, but your risk of issues goes down.

The Recommended Setup

  • Hardware: Stick to your plan for 3x Dell R740 or R740XD nodes. Ensure the storage controllers are in HBA/IT mode so ZFS can talk directly to the drives.

  • Networking: UniFi 10GbE SFP+ switching is perfect here. Dedicate one 10GbE link exclusively to the PVE Cluster/ZFS replication traffic so backup syncs don’t choke your VM network traffic.

  • The Cluster: Keep all 3 nodes in a unified Proxmox Cluster. This gives you a central pane of glass to manage everything and still leaves the option of doing HA if you really need it.

2 Likes
3

In my limited testing, HA in XCP-ng worked really well if a host went down, and even if it came back up later. No idea how it might react if the VM had an issue.

But since they are already running Proxmox, that makes sense because it is familiar.

4

Brilliant, thank you both so much for the replies.

Excellent video Tom, definitely enlightening.
I think I will go with the ZFS replication approach, as this is most likely sufficient in case of node failure.

Each node will probably have dual 1GbE and dual 10GbE, originally I was just thinking on bonding both NICs and create bridges on the bonds.
Is there a pros and cons for this approach compared to dedicating a link exclusively and not aggregating them on the network switch?

5

Once again if you don’t need the complexity then skip it. Of note, dedicate one of the 1G just for corosync. That is one place where you can have an issue because sharing the bandwidth at the physical layer could add latency to that connection and corosync is sensitive to that.

6

So, perhaps of a nuanced take (or maybe just a different perspective), if it’s of any value..

When I think of HA, the first thought that comes to my mind is “FT (Fault Tolerance)” or “HA”? and the second thought that comes up is (at what level?). For the record, I think previous posts may address this at many levels so please feel free to disregard this post, sprinkle with go-away dust, or a combination.

So let me clarify if folks are still reading.. The distinction between FT and HA I borrow from VMware, where 2 VMs are redundant at the OS level in some way. FT, of course, locksteps CPU threads, RAM content, I/O, etc. so that if one node goes down, it’s not suppoed to noticeable. There are requirements on affinities, vmotioning (in the vmware world), etc. Ok - so that’s if an app just can’t go down, even for a very short time.

HA has had downtime in terms of accessibility so that a secondary node comes up when the primary fails in an active/passive type of configuration. That’s fine for SLAs guaranteeing uptime at 99.n% and there are of course a few tradeoffs vs full FT.

However, there’s a third option, which is where the “at what level” comes into play, which is asking “what” needs to stay online. For example, if it’s a SQL server, clustering is available at its own level where critical DBs stay up either on shared storage or via log mirroring, and the application / web server is what’s OK to lose for a few minutes.

Of course, redundnacy is what’s behind all of it - at the storage (hw and fs), I/O, compute/app-tier, etc. It’s just what’s appropriate based on need.

So.. I probably just went off on a tangent in terms of what OP is looking to do, and the simple approach may be fine. I just wanted to offer that there are some nuances in terms of what approach I’d take from an architectural standpoint prior to applying a technology solution.

Hope this adds something, no problem if it’s different or too out in left field for anyone :slight_smile:

7

Hey all, thanks again for the feedback and my apologies for the delayed response.

I like David’s thinking of what fault tolerance is acceptable.
Obviously never experiencing any kind of downtime is most ideal, but also gets really expensive investment wise.

I think in our case, if disaster really strikes and takes down the node where a critical VM is running, we would be okay-ish with the ~5 min data loss between last ZFS replication for the VMs.

As for the applications themselves, it might be a really good idea to check out their options for critical data to be kept and avoid potential loss for databases.

So the ZFS Replication option for handling the “HA” of hardware level stuff, seems to be perfectly fine for me as of right now.

I really appreciate all the feedback and suggestions, thank you all very much.

Any suggestions for hardware to run Proxmox Backup Server on?
My understanding is the best practice is installing it directly on hardware and have it handle it’s own ZFS pool for backup destination storage, and possibly copy the backup data to a second target destination like a network share or another remote PBS instance?

Best regards

1 Like
8

You can run PBS on bare metal or as I do on TrueNAS

And yes, backing it up to another PBS instances is ideal.

I have a hardening guide as well

9

Thank you Tom. That was very helpful and enlightening.

May I ask why you took this approach compared to bare metal installation of PBS?

10

I use the TrueNAS for other functions as well. It also makes setting up, managing & maintaining the drives much easier.