So I’m running a dumb setup where I’m running proxmox as a vm ontop of truenas scale. I’ve been sticking with 12.2.2 for a while now because I heard about the issues involving pci-e passthrough and finally decided to try upgrading to 12.3.3 seeing they in theory patched it. My setup currently has me passing through 2 NVMe drives that are in a ZFS mirror inside of proxmox. But after the update to 12.3.3 it seems they have put restrictions in place to prevent you from being able to pass through NVMe devices. No idea if there is a work around. Tried to do a little googling but the only results I found were from the passthrough bug effecting people passing though gpus before the bug was fixed.
I did some testing before I put the proxmox vm in production for my home lab and I found between the zvol bugs and bad performance running zfs ontop of a zvol running on zfs that the best solution was to do passthrough and let the vm do its own storage. So I’ll either need to come up with a work around to let truenas scale passthrough my NVMe drives, or stick with 12.2.2. I was wondering if anyone might have any ideas on this.
I’d have a few ideas, but they’re probably not what you want to hear.
But first of all I would like to ask you a few questions about the use cases for such a setup.
What do you want to use Proxmox for? LXC containers? VMs? Both?
What do you want to use TrueNAS Scale for? VMs, apps (TrueCharts Containers), NAS functionality, or all of them?
One general answer in advance: In all scenarios I can think of, where you absolutely need to install both products on a single physical server, it would be better to do it the other way around, i.e. install Proxmox on bare metal instead of TrueNAS Scale, and then virtualize TrueNAS Scale on top of Proxmox.
Its primarily a file server. My reasons for doing this was I use to be able to shut down my file server on the hottest days of the year to save on energy and not bake myself as much but that recently changed. I can’t turn off my file server anymore. The work around was to replace one of the nodes in my Proxmox cluster with Proxmox virtualized on Truenas scale so I can still have the VMs I need to keep my network running without needing another server running as a trade off for needing my file server on 24/7 now.
But thanks for the suggestion, I guess I’ll see about asking over on the Truenas forums and see if anyone has any suggestions there.
I understand your train of thought, nevertheless I would still solve it differently.
If these VMs are able to run without any significant performance penalties as nested VMs on the NAS, wouldn’t it still be better to follow my strategy, and maybe use the second server exclusively as a backup target? Or maybe you could even ditch Proxmox altogether, run TrueNAS on both servers, and running the VMs directly on one of the two TrueNAS servers, or distribute them over both, and use the respective other one as a backup target (ok, in the latter case both servers would have to be always on again).
My current solution only has 1 server running, Proxmox can function with a single node online as long as you don’t need to make any changes. Plus I need the features of Proxmox mainly migration. If I can’t find a solution for the passthrough issue, I might have to go looking for an embedded solution that sips power as my 3rd Proxmox node instead of using my Truenas box.
