XCP-NG 8.3R2 USB Passthrough speed issue

I wasn’t sure where to ask this, but was hoping someone here might be able to help in some way.
I have a fresh install of XCP-NG 8.3R2, I have a single Debian VM running Docker, and a Frigate container. I have passed through a Coral USB TPU, which is connected to a USB 3.0 port. The passthru looks correct, I can see it at the Debian Level, and Frigate has access to it. The inference speed is very slow (120ms).
I’ve tested this before in the past on XCP-NG 8.2, and noticed a simlar issue. I had also tested it on Proxmox, with the same hardware, and I was getting the normal inference speeds (~10ms). So this seems to me that its specific to XCP-NG, and was curious if anyone else had had a similar issue, or had some advice on something that I could do to troubleshoot it or anything that I should be looking at?
Any help would be great, I am going to retest with Proxmox today just to doublecheck that.
Thanks,

Joe

I would cross post in Home | XCP-ng and XO forum

After posting this, I realized I did not check the usb speeds on the VM itself.
So where on the XCP-NG machine I see its at 5000M speeds, on the Debian VM I’m seeing this:
eppo@docker-test:~$ lsusb --tree
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/6p, 480M
|__ Port 1: Dev 2, If 0, Class=Vendor Specific Class, Driver=usbfs, 480M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=uhci_hcd/2p, 12M
|__ Port 2: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 12M
eppo@docker-test:~$ lsusb
Bus 002 Device 002: ID 18d1:9302 Google Inc.

Not sure why its 480M instead of 5000M.
Maybe I should try a different linux distro to test?

I see your reply Tom, I will do that, thank you for helping me out.