Lab pictures (extra characters)

1

In one of Tom’s live streams he mentioned maybe getting pictures from people showing their lab system, so here I am to start it off:

mini-lab-v2
mini-lab-v22000×1500 277 KB

Left to right, top to bottom:

VMware vSphere 1, 2, and 3, then Truenas Scale
XCP-ng 1, 2, 3 plus 2 more as yet to be determined.

All are HP T740 from Ebay, the top few even have a warranty. The bottom 5 were sold as BIOS LOCKED so they were cheap and in a lot of 5, so I bought 5. Supermicro dual 10g cards installed in the t740, as well as an a+e Intel i226 card glued to the back of the case. Hypervisors have (generally) 64GB of RAM, the exception is the one that has 32 written on it. The spare two are 16GB and 8GB respectively. The Truenas Scale is a Topton model N18 with n100, 16GB, two left over emmc drives for system, and 6 old WD Red NAS SATA drives in 1TB each (RAIDZsomething, I forget).

I am going to start migrating my lab VM’s from the old big XCP lab to these smaller machines now. Going from 400+ watts with the big lab down to about 200 watts for what is really 2 labs.

Not pictured is a Mikrotik CRS326-24s+2q+rm which is 24 10g SFP+ ports and 2 40g QSFP+ ports. The QSFP+ are broken out into 8x10g for the servers. The only thing I don’t really like is the n100 board has a 10g copper, so I have a wicked hot 10Gtek 10g-base-t module in the switch, CAT6a shielded between them (cable I had from other projects). There are some DAC cables in there as well, and may move at least one of the labs over to multimode in the near future, mostly because I’ll have it back when I shut down the big lab. Also not pictured is an old Cisco 2960s-24 with POE+ for a little 1G use, spilt into two vlans. This needs to go eventually, not doing much that needs POE+ right now.

What’s missing? IPMI/iLO, it comes in handy once in a while. I am waiting for a NanoKVM to arrive, I’ll play with that and see where the project is, and check on the possible security flaws that others have brought up.

Somewhere I have the big lab in a picture, not sure where it is right now.

2 Likes
2

Looking very clean and cool! Thank you for sharing.

What CPUs you have in T740s?

3

Found this post in a buffer, thought I hit the go button…

They are small, low power devices, AMD V1756B (4c/8t).

An 8 core, 16 thread AMD mini-pc would be about ideal, but you end up at around $500 each by the time you put in decent storage and 64GB of ram, and then only get one or two 2.5gbps connections.

I was lucky, that bottom shelf was $30 each (BIOS locked), a $25 programmer and some digging fixed them. Add in another $160 each for the three xcp-ng hosts (64GB DDR4, 256GB SATA m.2, dual 10gbps card) and you have a fairly potent small lab at 30 watts each at idle.

If you are running a heavy load on your lab all the time, then Enterprise servers are probably the way to go. Yes they idle at 100 watts, but at 50% that won’t be jumping up to 200 watts, more like 150 watts. At 50% load, these t740 will jump up past 60 watts, still a power savings, but not really designed to run at this level 24/7

I think one more of these will get a “big” drive installed and be used for a second pool. Then I can back up VMs to this second pool like a disaster recovery lab.

1 Like
4

So you’d say at 30 watts each idle and about 60 on load these 8 devices consume in total around 240 - 480W? Cool, now I don’t feel too bad about my setup :slight_smile:

I have 2 x HP Z840. Each consumes around 200W under my typical moderate load.

One thing I’ve come to realize a while back: there’s no magic bullet on saving power consumption. For the same total CPU power it’s roughly the same with small or big devices.

5

Well, here’s mine. Not nearly as neat as yours. On the top shelf I have a N100 pfSense box and a VLAN aware wireless access point. On the second shelf is my 30 port managed switch. On the third shelf is my second Proxmox node (HP Elitedesk mini G9), a raspberry pi based ham radio repeater (PiStar) and the Ring alarm base station. On the fourth shelf is my third Proxmox node (N100 NUC device) and my Synology DS220+. The next shelf and the bottom shelf are just storage for parts, etc. The 6th shelf has my primary Proxmox node (Ryzen 5 Pro 5650GE/node 304 case) and the cable modem. On the floor to the right is my HP Z640 which is my fourth Proxmox node and just a sandbox server really, where I experiment. It is mostly left turned off. On the floor to the left is an Ecoflow Delta 2 power station and small UPS. The UPS is plugged into the Ecoflow. On the window sill is the T-Mobile Home 5G device (backup WAN). My primary Proxmox node runs virtually all my services: Wordpress, TrueNAS, Nextcloud, Vaultwarden, Home Assistant, Discourse, Paperless-NGX, etc. My second Proxmox node is running my K3S cluster, my third Proxmox node runs ansible and a debian NFS server as a backup destination, and my fourth Proxmox node is where I am experimenting with things like Rocky Linux with the Houston UI, Unraid, and Debian with Cockpit and ZFS (sort of a Debian re-spin of Houston). All in with the HP Z640 turned off it draws 180 watts at the wall.

my-homelab
my-homelab1920×2550 380 KB

1 Like
6

My NAS sits in my office, as it doesn’t run all the time.

500×319 41.6 KB

Just got done upgrading / building the mini-itx machine at the top to bring bring everything up to 10G. It’s pretty overkill for OPNsense:

  • Intel 265K
  • Asus B860-I Gaming Wifi (Only Mini-ITX I found with Intel LAN)
  • 32 GB DDR5 6000
  • 1 TB MP600
  • SF850 PSU
  • Intel X550-T2