TrueNAS 24.04.1.1 (Virtualization PCI Passthrough)

Just as a heads up to the community – I get an error on the most recent version of TrueNAS Dragonfish (24.04.1.1) where I cannot pass-through my GPU to a TrueNAS virtual machine (nVidia Quadro P1000).

I opened a bug report, which was accepted NAS-129498. The error output I got for my specific issue was:

Error: Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 182, in start
if self.domain.create() < 0:
^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/libvirt.py", line 1373, in create
raise libvirtError('virDomainCreate() failed')
libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2024-06-10T01:55:16.481119Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:03:00.0","id":"hostdev0","bus":"pci.0","addr":"0x7"}: VFIO_MAP_DMA failed: Bad address
2024-06-10T01:55:18.338027Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:03:00.0","id":"hostdev0","bus":"pci.0","addr":"0x7"}: vfio 0000:03:00.0: failed to setup container for group 37: memory listener initialization failed: Region pc.ram: vfio_dma_map(0x55a96c4e59d0, 0x100000, 0xbff00000, 0x7f5fdff00000) = -2 (No such file or directory)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 198, in call_method
result = await self.middleware.call_with_audit(message['method'], serviceobj, methodobj, params, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1466, in call_with_audit
result = await self._call(method, serviceobj, methodobj, params, app=app,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1417, in _call
return await methodobj(*prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 187, in nf
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 47, in nf
res = await f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_lifecycle.py", line 58, in start
await self.middleware.run_in_thread(self._start, vm['name'])
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1324, in run_in_thread
return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1321, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 68, in _start
self.vms[vm_name].start(vm_data=self._vm_from_name(vm_name))
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 191, in start
raise CallError('\n'.join(errors))
middlewared.service_exception.CallError: [EFAULT] internal error: qemu unexpectedly closed the monitor: 2024-06-10T01:55:16.481119Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:03:00.0","id":"hostdev0","bus":"pci.0","addr":"0x7"}: VFIO_MAP_DMA failed: Bad address
2024-06-10T01:55:18.338027Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:03:00.0","id":"hostdev0","bus":"pci.0","addr":"0x7"}: vfio 0000:03:00.0: failed to setup container for group 37: memory listener initialization failed: Region pc.ram: vfio_dma_map(0x55a96c4e59d0, 0x100000, 0xbff00000, 0x7f5fdff00000) = -2 (No such file or directory)

Host ID: 6fe6aff0f8449635fadb0f9484a40e4a72f0a29267c3652574209dcd29921f79

Session ID: ec24720f-2a6b-274b-38ba-c30be069b3fe

They marked the upcoming version 24.10-ALPHA-ElectricEel as the fixed version, so I’m really hoping that will work.

1 Like

The support team marked my ticket as a duplicate of NAS-128544, which seems to indicate that this isn’t an actual issue/bug per se, yet more of my system is complaining that there isn’t enough free memory.

When I created the VM, I did get a memory warning, as I had requested the VM to have 24GB of memory, and the system (at the time) only had 10 … although it started, it did so with a warning. I got the error when I tried to attach the GPU as a PCI Passthrough device.

The PCI Passthrough device (GPU, in my case) will hard fail if there is a memory error. Rebooting the NAS so the VM (with GPU) starts up before ZFS has gobbled up all the extra RAM for ZFS Cache resolved this issue.