I’m hoping to refresh my home lab this year, and was hoping to get some advice/feedback from everyone here.
Currently I have 3 systems running.
E3-1271 v3 with 32GB RAM (XCP-NG)
Dell T610 Intel(R) Xeon(R) CPU X5675 with 98GB RAM (XCP-NG)
Intel(R) Core™ i7-2600S CPU @ 2.80GHz with 32GB RAM (TrueNAS Core 14TB 8TB Free)
The top two running XCP-NG are what I would like to replace.
I was thinking about getting 1 or 2 (depending on pricing) Minisforum ms-a2 using local storage for VM’s, and backing up to my Truenas. Eventually replacing Truenas system with something faster to use as shared network storage for the VMs.
I am currently running these services in Docker containers:
Immich
n8n
nginx proxy manager
grafana
influxdb
prometheus
uptime kuma
plex
Frigate
compreface
double-take
ollama
open-webui
And the following VMs:
Bitwarden
Home Assistant
Mysql Database server
Ubuntu (minor services on it)
Ubuntu (Nextcloud/Bitwarden)
Some of the reasons why i’m looking to refresh.
lower power consumption
Solidify my Frigate instance, have had issues with inference speeds while passing through my coral usb. Thinking about putting M.2 versions in these new servers.
Would like to give a little more resources to my Home assistant VM
With 2 Identical systems, the pool will actually work correctly and I can migrate vm’s across them and perform updates easier.
Overall stability
Working with Frigate and home assistant recently has sparked my interest in the upgrade. Based on recommendations from the Frigate site, it can run flawlessly on a $150 mini pc, so that is also an option. Anyone feel it is best to keep that app separate?
I’m just looking for overall opinions on this approach, what would you do if you were in my situation? Any flaws in my logic, or things I should take into consideration?
I have not used any of the new Minisforum PC’s but Wendell from Level1Techs seems to like them. I don’t run Frigate so I don’t have an opinion on that.
I have read that the MS-A2 will not support ECC memory. Are you OK proceeding without it? For the price you will pay for that box, I would think you could build a socket AM5 machine for less money and still have ECC memory. The only pluses for the mini’s forum (in my mind) are size if you are space constrained, and the included 10gbe networking.
Not having ECC is not a deal breaker for me.
You do have a good point, I could also go and build and AM5 machine.
To be honest, I feel like these mini PCs are sort of magic when it comes to power consumption. My current PC is running a 5800X and using 131W. I’ve seen some of these mini pc’s pulling <50W.
This may be a good opportunity for me to see what goes into building a power sipping server. Knowing what components use the least power at idle, but can deliver some processing power when I need it.
Is it still a bad idea to use processors with P and E cores with XCP-NG?
If anyone knows some good resources to learn how to pick power efficient parts, please let me know. Or If there are some current go-to cpu/motherboard combo’s to check out, that would be great as well.
Thank you
Money, that’s what goes into them. I spent a lot of time looking around to mini-size my lab for power/heat/noise… It will not be cheap. The best I was able to find was 8c16t AMD processors with up to 96GB of DDR5, cheaper if you go back a generation and get DDR4 up to 64GB. But you are still looking at over $600-$700 each and you may be stuck with Realtek 2.5gbps connections.
For now I went a different way, bought a lot of five HP T740 with BIOS password lock, working through unlocking them now. These will require a cheap SATA m.2 drive for system, and $100 each worth of DDR4 to get up to 64GB. They will also need a network card, probably 10gbps for about $30 each. XCP-ng is the target OS.
Why those “mini” PCs? They seem to be working fine for light lab use with VMware, so should perform about the same for XCP-ng. It’s a lab, so not hard use and also not constant use, I normally tear it down and start over at least once a year.
I very recently built a server based based on socket AM4, for two reasons: First, the price and availability of ECC ram for the AM5 socket. ECC memory for AM5, when you can find it, is almost 2 times the cost of memory for AM4. Second I kind of became enamored with the 45Drives/45Home Labs HL4 and HL8 storage offerings. So I decided to create an HL8 clone. I used the same motherboard (Gigabyte B550I Aorus Pro AX), a Ryzen 5 Pro 5650GE that I found new on ebay, 64 gb of ECC RAM, a Corsair RM Series RM650 PSU, an X520 10gbe SFP+ NIC, and a Node 304 case. I loaded it up with 8 Samsung SM863A enterprise SSDs and I am running a litany of services on it. It idles at 40 watts. I have Proxmox as my main OS, TrueNAS Scale virtualized and I am running 5 VMs and 15 or so containers. It cost me (without drives) ~$900 to put together. Not saying its a better option than the Misnisforum machines, but just to give you an idea of what’s possible if you DIY.
I think it might be best to try a custom build for the hypervisor.
Based on some reading I’ve been doing, something built around the Ryzen 5700G, might be a good fit for power vs performance?
Maybe pair it with a MSI PRO B550M PRO, based on what I was reading?
Curious if anyone has had any experience with a similar hypervisor build?
For Frigate, I think i’m going to pick up an N100 mini pc as recommeded by the developer and just call that a day.
Curious if anyone has had any experience with a similar hypervisor build?
Sounds almost exactly like the build I did, except I used a Ryzen 5 pro 5650GE. Just bear in mind the non pro Ryzen 5000 series APUs do not support ecc memory
I was trying to find a mini-pc with an 8c16t Ryzen processor when I was going through this, for 3 computers with at least 64GB of ram and some level of storage was coming out above $900usd and I just had to stop. But this was my preferred processor for a low power lab (57xx or 58xx).
Found a deal on some BIOS locked HP t740 (4c8t) that I’m already familiar with using (my vsphere8 is running on 3 of those) and just went forward. Once I got the bios unlocked, I bought the rest of the stuff needed. Used dual 10gbps NIC, a 2.5gbps a+e i226 NIC, SATA m.2 drive for system, and 64GB of ram for each (3 PCs). I think XCP-ng will be fine on this for what little performance I need out of my lab. Still draws more power than these newer mini-pc, but I just couldn’t get all the way there with what is available in the cheaper mini-pc territory. I’m in for around $350-$400usd for 3 plus 2 at stock configuration that will get used for “something” later on. When the lot of computers is $30 each, you kind of buy the entire lot and figure out the extra bits later.
Crucial Pro 64GB (2 x 32GB) 288-Pin PC RAM DDR4 3200 (PC4 25600) Desktop Memory Model CP2K32G4DFRA32A
I was thinking about just putting 2 Samsung 990 Pro M.2’s in it. Installing onto them in a mirrored configuration, and using the rest for VM storage to keep things simple. In the future when I upgrade my NAS I’ll try using Shared Nas storage.
Curious about the ram, will choosing faster ram actually result in noticeable speed increases for server type use cases?
Any comments on the build would be appreciated, just trying to make sure there aren’t things I might have overlooked.