I am planning to expand my home network and move it (or at least parts of it) to 10 GB. Currently, I have the following setup:
Laundry room: in-wall media enclosure (see picture here):
Modem (Spectrum)
Switch (Ruckus ICX 7150-C12P)
Router (AliExpress mini PC) running pfSense. The device also runs pi-hole as well as some essential homelab services
Unifi CloudKey
Lutron bridge (For Lutron Caseta light switches)
The builder terminated all cables into this enclosure. All rooms are equipped with Cat5e and I am also running a few cameras through PoE.
Office Closet: Homelab server connected via a single 1GB Cat5e cable hosting multiple VMs and containers (homelab stuff).
I am planning to extend my homelab with more kubernetes nodes, UPS etc and I came up with the following requirements for the switches from a port and connection speed perspective.
For the laundry room:
Device(s)
10 GBe
1GBe
Cameras (today)
5 (PoE)
Cameras (planned expansion)
3 (PoE)
Normal rooms
6
High bandwidth rooms
2
Router uplink
1
Server room
2 (fiber)
UPS
1
Future 2nd router uplink
1
Total
4
17
I am planning to run two fiber cables from the laundry room into the office closet so that I have the option to move to 25GB or 40GB in the (far) future, but I also read that fiber runs less hot, has better latency and consumes less power.
Office Closet (server room):
Device(s)
10 GBe
1GBe
Uplink to laundry room
2
Main server incl NAS
2
1 (management/IPMI)
Cluster Node 1
2 (for Ceph)
1 (management/IPMI)
Cluster Node 2
2 (for Ceph)
1 (management/IPMI)
UPS
1
Lutron Bridge
1
Unifi Cloud Link
1
Future separate NAS
2
1 (management/IPMI)
Future NVR
1
1
Total
11
8
I would like to future-proof it (as far as that is possible and makes sense with technology of course), but I also would like to have an energy-efficient setup that is as quiet as possible. Based on what I read here on reddit in the past days and weeks, my favorite brands are so far Unifi and Mikrotek, but I am open to additional ideas. I really like the Unifi UI and it would integrate nicely with the existing NVR and Camera setup. Unfortunately, they do not seem to have an affordable PoE switch with enough ports for 10GBe. Based on my research, I have the following switches shortlisted:
Maybe y’all have an even better idea how to achieve the setup. As you can see in the picture, the space in the media enclosure is rather limited, so the setup needs to fit in there. A standard 1U switch would fit.
Compared to MikroTik are more complicated to setup, really comes down to your knowledge of them. Since you already have the Cloudkey I would stay with a UniFi setup.
If budget is a constraint, there are various fanless 8+2 port PoE switches you can buy, instead of that 24 port PoE. I use Netgear switches, rather than buying a 48 Port PoE I bought it without PoE and just added another 8 port PoE switch. Handy now as various devices now need PoE+ etc.
We never have used Ubiqiti support but I have heard it’s not great which makes sense because we sell so much consulting for their products. But deploying thousands of access points & switches I don’t see any issues with functionality of those devices when properly configured.
I’m a relative network noob, so pardon me if I’m missing the point. I know you want to “future proof” but why two 10Gb links to each device you want on 10Gb? It’s nice to have redundant connections, but if you’re trying to contain costs, I’m not seeing the benefit.
As for the IPMI, one option is to put them all on a physically separate switch. Doesn’t even need to be a managed switch. Whether that is even connected to the rest of the network is an open question. Physical isolation can be nice. That also cuts down the number of ports needed on your 1Gb portion of your network.
You might want to consider putting the web configuration interfaces on a separate vlan.
Not sure why you’re looking for a 10Gb switch with PoE. My own inclination is to have one switch with 10Gb and another that has the PoE devices.
Without redundant connections, you’d only need one Unifi Aggregation switch in the office:
NAS 1
NAS 2
Cluster node 1
Cluster node 2
NVR
One more port to connect to the rest of the network and two ports still available.
Edit 1: because I missed your plan to put the Aggregation switches in the office
Edit 2: Put a 10Gb switch and a 1Gb switch in the office. One 1Gb link from laundry room to office. And one 10Gb link (if 10Gb is needed) from switch with the cameras in the laundry room to the NVR in the office. Keeps camera data off your regular 10Gb network.
Check out the Unifi EnterpriseXG 24 it will give you all the 10G ports you need as well as some 25G ports for your future proof and at $1299 it’s fairly reasonable.
My vote on the MikroTik switches as well. Steep learning curve but it gives you so much control on how you want it done. There’s even command line if you like being a “Cisco” guy.
Thanks for all the input so far! Another challenge I need to think about is space/size and heat dissipation. My media enclosure that will house the 24-port PoE switch is 14 inches wide and the switches are between 10 and 11.5 inches deep. I will continue to use the fans you can see below, but cut a whole in the door and have the bottom one suck “cold” air in and the top one push warm air out. See a picture of the current setup (with an ICX7150 switch) below:
I thought a little bit more about my setup and came up with a new idea: I do not want to cram in the switch into the enclorure, where there is not really enough space when taking Ethernet cables to the front and power cords at the back into account. My new idea is to remove the switch from the enclosure and hang a 6U or 9U closed rack above the enclosure. The dryer “vents” to the outside of the house, so the room is not more humid than then rest of the house. This way I can move the switch in into the rack and also add a UPS. Another idea is to then use the added space to properly terminate all cables into a patch panel and then route the cables from their back out of the enclosure through the drywall (using this) into the rack. Does it make sense overall? Should I put another patch panel into the rack or would you go from the enclosure panel directly into the switch?
I recently had FTTH installed. In preparation for my ISP link going up to 1.5 Gbps symmetric, I added some cheap TP-LINK 2.5G cards to my pfSense router and one wired desktop and one wired NAS in my home lab.
Also needed at switch capable of 2.5 GBe so I bought a $99 VIMIN from Amazon Canada. All is working well and the performance is dazzling.
The Netgate folks don’t seem to like the Realtek chips but the latest drivers work very reliably.
So I would recommend you think about 10Gbit because costs really do escalate and 10GBe is power hungry and generates tons of heat. So I would only go 10Gbs with fibre SFP ports. Also 10 Gbs is very difficult to achieve over anything less than Cat 6a cable and the pfsense needs quite a bit of horsepower to drive the PCie channels at that speed.
But good luck with your project if you do go down the 10Gb path.
Thanks,
I will only do 10GB over fiber or DAC. Currently the Unifi Enterprise 24 PoE is the frontrunner, because that would mean I can connect a 2.5GB Wi-Fi AP somewhere down the road.
Hi Mark - I believe that Google’s 5GB and 8GB offerings provide a 10Gbase-t connection, so there is that.
I am considering a) getting the 5GB Google fibre and b) one of these switches Cheap 8-Port 10Gbase-T Managed PoE Switch the Hasivo S1100WP-8XGT-SE which are pretty reasonable on power (article says only 11.2W with one 10Gbase-T linconnected. My one server with SFP+ ports I will run 10Gbase-T SFP+ modules as needed to connect.
I just recently went down the 10G path in my home lab. I used a MikroTik CRS354-48P-4S+2Q+RM its the first non Ubiquiti switch that I have used that detects 24/48v for devices like Nanostations, Rockets etc. and I broke out the QSFP into 4x 10G links to my NAS, Surveillance, ESXi and Proxmox machines via DAC. Next will be a second DAC breakout cable for my pfSense box and another dev host running Proxmox with 2 to spare. I have 13 cameras running into a BlueIris with plans for several more. I did this so devices would not have to compete for bandwidth on the servers. But if you look at the end user devices that are accessing the services I would suspect that you are not fully saturating a 1G port. My wife on her design station is the only device on the network here that can achieve saturation of 1G aside from the PtMP on the roof that acts as the receiver for the security camera CPE units throughout the mobile home community. If you need a ton of 10Gb MikroTik makes a fantastic switch with 24 SFP+ (CRS326-24S+2Q+RM) ports and if you already have some MT gear in your env then you are familiar with its ease of use. Once you know one MT you know them all (swos being the exception).
