What's next for my homelab?

Computer Hardware & Server Infrastructure Builds
1

Hi everyone,

first, I would like to thank everyone who posts so many cool tips and experiences here. I have started my homelab journey about a year ago and am pretty happy with the setup for now. This is what I am currently running:

  • Topton mini-pc: with four 2.5GB ethernet ports - 4-core Intel N5105, 16GB RAM, 500GB NVMe - I am running proxmox on this. On proxmox I run pfSense (router and firewall), pihole (ad-blocker) and a VM with essential docker containers (authentik, diun, homepage, homebridge, npm, portainer, scrypted, uptime kuma)
  • Main server: Fractal Design Node 804 (AsRock Rack EPC612D4U, Intel Xeon E5-2640 v3, 250 GB SSD (for Proxmox OS), 1TB NVME (for VMs), some HDDs for NAS. I am also running proxmox on this. On top of proxmox I run TrueNAS as (guess what) my NAS. I am passing through the RAID controller to TrueNAS. In addition, I run my pihole failover (HA with keepalived) as well as two VMs with docker containers. One for the *arr stack and the other for internal services, e.g. audiobookshelf, plex etc.
  • NAS Config: 3 drives. Two 4TB drives as RAID 1 and one single 18TB drive for movie and TV backups. All drives are 1-2 years old.
  • Switch: Ruckus 7150 switch with some Unifi Protect cameras and two 10GB ports. 12 port POE switch, which is enough for now, but with the cameras I would like to add all ports will be in-use.
  • Location: Everything except the Main server is housed in an in-wall enclosure in our laundry room. The main server is in my office closet, which is easily accessible and I can run additional cables there “relatively” easily.

I would like to develop the setup further to have HA for all my services as well as reduce the energy footprint. I am reading a lot about kubernetes and proxmox clusters, but I am not sure what to do next. From a requirements point-of-view I would like to do the following:

  • HA for all services
  • In the best case, the internet connection is the only one without a failover. I am too cheap to pay for two providers, but I have read, how I can run internet through my phone in case internet goes out (which it rarely does).
  • Minimal energy footprint
  • Small “space footprint”. I do not want to put a huge rack anywhere. I would have the space to put a 9U rack on the opposite wall of the wall enclosure where I house everything (except the server) today. This would mean I need to reroute cables.
  • 10GBit ethernet between my main machine (Macbook Pro M1 PRO) and the NAS.
  • SSD / NVMe-only environment. This would only be a mid-term “requirement”. I will not be able to replace the 18TB drive anytime soon with SSDs without selling my car. that drive is not even a year old.

Based on what I have read, I thought about this as my next steps:

  1. Short-term:
  • Bring macbook and main server to 10 GBit. Is dual 10GBit worth it for the server? The current switch only has two 10GB ports and I do not want to invest in hardware, if I do not need to.
  • It seems like I do not need kubernetes. It looks very cool in all the videos I can watch on youtube, but I do not feel I have a use case for it.
  • Bring in a second mini-pc to run a proxmox cluster. Could I combine the three machines into one cluster and with this have HA for pfSense etc as well? Or do I need to have three exact same machines to run a proper proxmox cluster?
  1. Mid-term:
  • Replace 4TB disks with SSDs
  • Decide on rack or not. I feel like I could run the ISP-modem, the router-box and the switch still within the wall enclosure and only run the other things in the rack. this would have the advantage to have everything in the laundry room.

I hope what I wrote makes sense and a few people can tear this concept apart and have some better ideas :). I am especially interested in rack vs no-rack and cluster setup recommendations. Thanks again for some tips.

2

Not too much to add but …

My view is the 2.5GB is a bit of a fudge, 10GB is what’s wanted, so I think waiting until this is viable financial option is best.

However, if you really have a lot of traffic on your network, then using a LAGG between switches and devices might bring some benefit, if only some failover. You don’t mention any backup solution, perhaps having Proxmox Backup Server might be worth a consideration, I had meant to look at it in the past myself but only did just recently. You can also use it as a backup for other stuff.

3

It’s a home lab, do you really need 10gbps? That said, I have 10gbps on my lab. I used a Mikrotik CRS309-8s-xxxxx which is all SFP+ 10gbps ports (can take 1gbps but you might need to force the port speed to 1gbps because auto detect may not work for all modules) and using fiber for the servers to connect.

I didn’t see much performance increase going from 1gbps to 10gbps until I reworked my storage from 4 drives to 8 drives in the array. And even then they are slower 3gbps drvies so not a lot more speed. I do get faster VM migrations from one host to another since it is only moving RAM.

My production system uses the same switch as a “top of rack” switch, I get 10gbps transfers when migrating VMs, but this is a brand new system with modern servers.

4

HA for pfSense with Proxmox cluster is tricky. You need the connection from the ISP plugged in both machines and making sure it maps to the right adapter, not to mention other issues. If you really need HA I suggest running 2 instances in fail-over mode, but again there will be slight complications with the wiring.

One thing I didn’t see in your setup is backups. You’re running pretty much everything in Proxmox VE, highly suggest you look into Proxmox Backup. Gives a bit of peace of mind knowing you can just restore either the full VM or per file restores.

My own setup for backup has 2 Proxmox Backup servers, each with 1TB disk. All VM machines are backed up on a schedule on PBS 1 and nightly there’s a replication to PBS 2 for redundancy since each machine only has 1 disk.

5

I have pfSense running in HA on my Proxmox Cluster I used managed switches to provide a link between the modem and Proxmox on two dedicated VLANs so that should one switch fail the other still has a link to the modem and each Proxmox node is connected to each switch and only the pfSense VM uses the assigned WAN VLANs.

2 Likes
6

Sorry for my late response. thanks for sharing your thoughts!

So you have a physically separate Proxmox backup server? So far I backup all VMs and containers via a daily job onto my NAS (which is virtualized on proxmox as well).