TL;DR - I’m looking for a cost and power efficient way of hosting 5-8 Talos Linux machines in my lab. The cluster would not be mission critical, but the intention is to configure everything as if it was. Should I buy one larger machine and use XCP-ng to carve out VMs for my nodes, or is it worth the hassle to get a bunch of NUCs (or other hardware)?
Current Setup
Today, server wise, my lab contains a TrueNAS machine (used solely for storage) and a Ubuntu box which hosts a handful of services, some of which mounts NFS shares from the TrueNAS server. The setup is serving me well, but in the name of educating myself I want to replace the Ubuntu machine with a Kubernetes cluster.
But why Kubernetes?
I have been working with Kubernetes for several years, but as a consultant I’m often brought in to fix a specific issue, or set something up and then hand it over. It’s fun and I’ve learned a lot, but with this project I want to setup and maintain an environment that could be utilised both for demos and be part of my portfolio.
My thoughts right now
My Ubuntu machine is not particularly beefy – it has a Ryzen 5 3600 and 32G of RAM – but it gets the job done. I understand that both options (virtualisation and having dedicated computers) adds overhead and that I need to account for this. However, I’m not interested in buying a monster of a server for my new project, while my gut feeling tells me that finding 5-8 small form factor computers, with suitable specs for my project, will be difficult and expensive.
Furthermore, I don’t think I can justify buying enterprise grade hardware. It would be cool to spec out a Supermicro machine with an Epyc CPU, but since the machine(s) I’m building will not have anything critical on them (everything is configured as code and the data is stored on the TrueNAS) I don’t think it’s worth it.
My current plan is to build a system around a Ryzen 9 5900XT and 64G of RAM. I could run XCP-ng and virtualise 5 VMs (3 CP + 2 W). I’d assign the CP nodes 2 CPU / 4G RAM each and the worker nodes would get 4 CPU / 8G RAM. Should I realise that I need more compute down the line, I could buy a NUC or two and extend the project to include both virtualised and physical nodes.
Help, please?
What would you do?