20 slow cores and 4 fast cores, how to get the most out of my homelab?

Computer Hardware & Server Infrastructure Builds
1

Hey everyone! First time poster so hopefully I don’t mess up too much, please let me know so I can move/edit/delete the post as needed.

I have two main systems for my homelab currently. Looking for advice on suggested use of my two systems before I get too much further. Whether it’s sell, upgrade, downgrade, or just just send it, I want to see peoples thought processes as I’ve been racking my brain on this for awhile of the “best” use of everything.

First system, currently running Proxmox, is a HP G4 Z4 with the following.

  • Xeon(R) W-2123 (4 core, 8 thread)
  • 128Gb DDR4 ECC
  • AMD RX 580 (I think, it’s an 8GB card)
  • Sata ssd storage, 512 for boot and ISO, 1tb for VMs
  • 10gb SFP+ NIC

I’ve been using the above as my main virtualization server, to mainly test and spin up random things. Currently has a kali box, a Wazuh VM, and a VM running docker for random testing. Also have some other random VMs that are spun up on demand with no real demanding tasks. Have been looking into hosting some game servers for friends.

The other system I recently picked up for what I think is a good deal. A dell T420 for $200 with 8 drive sleds. It’s running TrueNAS, here’s the rest:

  • 2x Xeon(R) CPU E5-2470 v2 (20 cores, 40 threads)
  • 128Gb of DDR3 ECC
  • H310 which is in IT mode
  • also added a 10Gb SFP+ Nic

The Dell also came with a 4 port NIC that I’ve removed for the time being. This system was purchased to serve storage needs, but with having 32 more threads than the HP box, I feel like I could be leaving some lab experience on the table.

I’m open to suggestions on how to use this available compute or possible upgrade paths with the hardware to improve use. My current thought is trueNAS on the Dell, and run nextcloud and Syncthing on truenas. Then the HP can remain as the proxmox host for whatever else, possibly upgrading the CPU down the road to a w-22XX series.

Curious what the community would suggest in this case, as I’m only $400 in with both systems total, not including the SFP+ adapters or SSD/HDD. The dell just seemed like a good deal on a NAS solution, but at 20 cores, even if old, seems like a lot to leave on the table. Figured some “well if I were you” might help spark some creative ideas, or help prevent some headaches. Thanks for reading and looking forward to this thought experiment!

2

I wouldn’t worry too much about thread count. Hypervisors can usually “over subscribe” CPU cores with almost no negative effect. I often assign more cores than my machine actually has. Its memory that is much more sensitive to how much you assign. Some hypervisors like proxmox can use something called “balloon” memory but that isn’t really the same as over subscribing. My original Proxmox host was a HP Z640 with an E5-2690v3 CPU with 12 cores and 24 threads. My current Proxmox host has a Ryzen 5 Pro 5650GE CPU: 6 cores and 12 threads. It is way faster than the E5-2690 machine. I am currently running 6 VMs: A TrueNAS Scale VM with 6 cores and 16 GB of memory, and 5 Debian 12 VMs running things like wordpress, Discourse or as dedicated docker hosts. the forum is assigned 4 cores, the two Wordpress VMs have 4 cores each, and the remaining two have 2 cores each. That’s 22 cores assigned! And my physical CPU usage rarely gets above 10%, and it is sitting at 2% right now basically idling and running some ansible apt update playbooks. Inside the TrueNAS VM, I am running Nextcloud, MariaDB, Redis, Vaultwarden, Postgress, Nginx proxy manager, Portainer agent, and watch tower. In a different VM, I am running cloudflared, Heimdall, Uptime Kuma, another instance of watchtower, Portainer and Librespeed.

All of this is to say that you can run anything on either of those machines. I would likely use the HP Z4 as your virtualization host. Or maybe both of them, and virtualize TrueNAS or some other NAS OS on the Dell, Then you are close to having a cluster to play with, if you add one or two more pieces of hardware, like a mini pc, NUC box or old laptop that you have laying around.

3

If I were you I would run TrueNAS on the Dell in a VM and give it as much memory as required for RAIDZx (was it 1GB RAM per 1TB storage?) and pass the SATA controller or disks through to it. I would add a NVMe or SSD for holding the hypervisor and the VM vdisks.

Then I would put the VMs on the Dell that are storage access intensive.

At the same tie both hypervisors would use the TrueNAS system for shared storage so that you can have quick migration between the machines.

In this setup you still cannot really do quick migrations to shutdown the Dell for maintenance, as it depends on the Dell. But you can do full migrations including the vdisks, it will take along time though.

you would want to use the SFP+ interface just for the (non-routed) storage traffic. If you can use double interface cards you could reserve the other SFP+ port for migration traffic.

The amount of RAM sounds more than enough for what you have described.

If you wan to be able to migrate VMs between the machines you need to make sure that interfaces and VLANs have identical names on both.

I believe that since both hypervisors have different CPUs a hot VM migration wouldn’t work.

In general I would be concerned about power consumption, but it depends on where you live and who pays for the power. It may be better to build less powerful but more modern machines that deliver just the compute power you need at a much lower power consumption. Make sure to calculate the investment and the point of beak-even.

4

I could sell you a lot more if you were close to me, getting ready to dismantle my “big” lab for a mini lab. 3x 20cores (40 threads) and 128GB each, plus a Truenas with eight 2.5 inch bays. All with 10gbps cards as well and gigabit cards if you needed to swap.

Just waiting for RAM for the minilab so I can move a couple VMs over and take it down. 400+ watts idle down to around 100 watts idle.

5

@Louie1961 I hadn’t considered the fact that if I virtualized TrueNAS on the dell I’d be 1 more computer away from a cluster. That is a tempting idea I hadn’t considered! I’ll add that to the list for a later day whenever the next overhaul is, as I think that would be fun to learn and use.

@xerxes I’m glad you mentioned routing storage, as I was considering keeping the VM disks on TrueNAS on the Dell, and using the HP as the hypervisor. With how everything is configured I’d end up routing that storage (sorry Tom). I’ll work on getting my network mapped out and labeled now so it’ll be easier to see and work with what traffic is going where.

For the power consumption, including modem, gateway, POE switch, a Mac mini, rasp berry pi, the HP, and the dell it idles around 200 watts. Certainly higher than a modern computer, but some of the expandability and upgrade paths are much cheaper with this used gear should I need more resources. I do think moving to mini PCs for the home “production” is something I’ll add to my roadmap. With summer approaching all I can think about is how much heat this set up is also generating.

6

@Greg_E I appreciate the thought! For the time being I’m doing my best till I finalize the list of what all I plan on running on which systems, and then leave it be before deciding what’s needed or not.

New so I couldn’t mention 3 people in one post, I’ll be sure to look at some more posts on what the better way of handling so many responses is.

7

If you are routing the storage for virtual disks you will have unwanted performance impact. You really want to re-design that so that hypervisor storage backend traffic is not routed.

8

if you want HA clustering you need to clarify that the hardware setup of the 3 machines is compatible for that. As an example, if you had 1x Atom 2x i7 you would only be able to do hot migration between the i7 machines, not between the Atom and the i7. Although all are arm64 code compatible. Someone correct me if I am wrong. It is this way for ESXi vCenter.

9

XCP-ng might be able to do what has been called a warm migration, I don’t really know how that is different from either on and running or off (hot or cold).

But I’ve been told this warm mode can be done between AMD and Intel processors, going to find out soon I guess. (big lab is Intel, mini-lab is AMD)

10

Currently the hypervisor has local storage on the system for the virtual disks, but I think it would make more sense to move all the data to the one system and just letting the proxmox host do compute. I’ll have to look at how I want to do that, thinking maybe just a DAC between the to?

HA isn’t on the radar yet, but I hadn’t considered the compatibility issues. I’ll keep this in mind for whenever I start looking for more/new hardware.

11

Possibility of working with HA is why I always put 3 (identical) machines in a pool. I did play with it early on, but not currently using it in production or lab. Things like rolling pool upgrades/reboots also work “better” with 3 hosts.

Built my vmware lab with 3 hosts too and it seems like a good idea even though I don’t intend for it to be a long term lab. XCP-ng is long term since I run it in production and may want to test something before building it for production.

12

Ah this is a good point. I don’t have a whole lot going on yet where a reboot or upgrade would be that concerning, but being able to continue to work on projects as the reboots occur is compelling. I now see why things like the minisforum MS-01 exist, as you could cluster those together and also have SFP+ to the storage system. Some used SFF would probably be more cost effective for my lab though.

13

It’s also about the process, not as much about being able to continue work while the updates are happening. In production, I’m not going to be able to shut everything down to update XCP-ng, they don’t pay me overtime or to work on the weekends. So keeping services going during this process is just part of business. I should really learn how to schedule this, but I also like to be able to verify that the servers came back up and attached back to the system after their reboot.

And it works really well, attached listeners to our Shoutcast stream don’t even get kicked, I think there is nothing more than a glitch when the switchover happens. I’m going to need to monitor that when the next patches come in, just for fun.

To expand on this, in production you really need to have some sort of backup storage so you can move the VMs from main Truenas to backup so that Truenas can get updates. Local is fine for this, a second slow Truenas is also fine for this. XOstor might also be fine for this type of stuff since it gets updated with the system, but I haven’t bought it and haven’t figured out how to build it myself. Apparently you can build XOstor from sources, but XO will not manage it so you will need to learn the commands. It coould be useful for me in my small production system.

14

just want to make sure you noticed that @Greg_E said he uses 3 identical machines for HA, with emphasis on identical.

15

Variation in amount of ram won’t be a big deal, and even something like Scaleable v2 silver with v2 gold might not be a big deal, but I’d only do that level of processor difference while swapping out fewer to greater core count processors. I’ve been told this 10 to 20 core processor swap should be no problems when I get the funds in my work budget. Would also like to add another pair on 64gb ram modules to bring each host up to 256GB.

It’s going to be interesting to go from Xeon v2 to AMD v1756b as soon as my ram arrives, supposed to be delivered on wednesday. Hopefully this works out, or I’m a few hundred $$ in the hole.