Learnings from xcp-ng and TrueNas a hypervisors

A few months ago, I was entirely happy with my homelab set-up: a Ubuntu server running docker, and a bunch of docker-compose stacks to provide:

  • email server
  • three websites
  • photo gallery
  • database server for gnucash
  • jitsi conferencing
  • home automation (home assistant, esphome, app daemon)
  • file server (samba and nfs)
  • iSCSI server
    and probably some more I’ve swept under the carpet.

Anyhow, I stumbled across LTS’s Youtube videos on xcp-ng, Proxmox and TrueNas and thought “there’s something interesting to play with”, also know in our household as “breaking the email”.

Initially I decided on xcp-ng as my homelab, with TrueNAS as a virtual machine to provide file services, and a Debian virtual machine running docker to provide the services.

I discovered one big gotcha here, which is probably so obvious that it doesn’t appear in the YouTube advice. It’s a bad idea to make your xcp-ng dependent on a virtual machine it is running. But you kind of want to do that because xcp-ng isn’t really a filesystem manager. It supports zfs, although only weakly, if you want to use zfs snapshots and replication you have to do this from a ssh command line. I want to mount an isos filesystem from the TrueNas server.

Ultimately I decided I needed the hypervisor to have no dependencies on the virtual machines it runs. So I switched to TrueNas running on the bare metal, providing zvol disks and nfs exports to virtual machines. TrueNas don’t have anything like the richness of Proxmox or xcp-ng defining and running virtual machines. But it’s OK for me and my single-machine home-lab.

Actually, it’s not a single-machine home-lab because I also have a backup server that is woken and put back to sleep from cron jobs on TrueNas, and acts as a replication target.

I hope these experiences are interesting/helpful to others in a similar situation to me. I’m sure that’s a lot different to the professionals out there with racks of equipment. :slight_smile:

Adrian Stephens, Cambridge UK, retired software engineer and now full-time meddler and fettler