10%-40% Packet Loss When Somebody On Network Streams Video

Networking & Firewalls
1

TL;DR When somebody on the internal network starts streaming video, pfSense reports 10-40% gateway packet loss. Anybody seen this issue before?

This is a very strange fault.

Our hardware router died last week, so I replaced it temporarily with an Intel NUC running Proxmox Virtual Environment (v8.2.2) running the latest pfSense CE.

Long story short, everything works fine until somebody on the network starts streaming video (e.g. Zoom, Teams etc).

At that exact time:

  • A continuous ping starts to time out 10-40% of the time.
  • Large file downloads start-stop, start-stop etc.
  • The Gateway (Dashboard) reports 10%-40% packet loss.
  • The Traffic Graph (Dashboard) shows LAN traffic in, but intermittent WAN traffic out during stream.

The issue is repeatable: stop the video stream, everything returns to normal, start the video stream, packet loss begins again, rinse and repeat.

Audio seems to be unaffected.

Default firewall and NAT rules i.e. nothing allowed in, everything allowed out.

Two interfaces, WAN and LAN. No VLANs.

LAN: In-built NUC Intel NIC, port eno1 bridged to vmbr0 (default proxmox config), static IP 192.168.xx.xx/24, bandwidth: 1Gbs Ethernet.
WAN: External USB Adapter, port enx00e04c9689bc passed through to pfSense appearing as ue0, static IP 51.xx.xx.xx/30, bandwidth: 100Mbs FTTC.

Troubleshooting already done:

  • No memory or CPU issues showing in Proxmox or pfSense. Multiple GB memory free, during packet loss CPU at 2-5%.
  • Updated Proxmox and pfSense
  • Disabled / re-enabled hardware Crypto support
  • Disabled / re-enabled IOMMU (and passthrough)
  • Re-installed pfSense
  • Re-installed Proxmox
  • Connected directly to WAN, no issues.
  • pfSense Monitoring shows correlation between Quality > Packet Loss and System > Interrupts & System Util

Has anybody experienced this before?

Any help would be warmly received - I’ve been troubleshooting this issue now for 4 days straight🙁

2024-05-13_18-51-15
2024-05-13_18-51-151920×1200 279 KB

2024-05-13_18-52-35
2024-05-13_18-52-351920×1200 277 KB

2024-05-13_19-56-09
2024-05-13_19-56-092424×2854 454 KB

2

I would guess it is the USB interface, especially if it is a Realtek chip. I’d also install bare metal if you don’t need Proxmox for other things.

And if this still doesn’t work, try OPNsense, there is a Realtek driver package you can install once it is running, might help if your usb is a Realtek. You can install the newer Realtek driver for BSD into PFsense, but I don’t know the commands to get it installed.

2 Likes
3

I agree with @Greg_E. You have a big whammy going on with 1. Virtualizing pfsense and 2. Using a Realtek NIC and possibly 3. It’s a USB dongle.

I would highly recommend you switch to a physical firewall as soon as possible with the proper intel hardware.

4

@tictag, just because you didn’t mention it in your detailed description, have you disabled Hardware Checksum Offloading under System → Advanced → Networking?

5

100% agree - this is a temporary measure before our network upgrade coming in a few weeks, which includes a Fortigate 60F firewall. Just didn’t want to buy something to only then replace it in three weeks.

I have no idea what that is but right now, I’m willing to give anything a try!

6

Update: it was NOT disabled, but disabling it (and rebooting) had no effect.

One thing I have noticed, however, is that it is only video streaming OUT that causes this problem. Incoming video streams work fine.

7

What did you set the network driver to be for the pfSense WAN interface? I too had weird problems with a realtek 2.5G NIC on a Proxmox / pfSense VM. Setting the driver in the hardware options of the pfSense VM to E1000 brought stability back for me. ISTR that Proxmox tried to pick the VirtIO driver by default.

8

I probably have the same adapter, mine is an Anker 2.5Gbs USB Ethernet Adapter, uses a Realtek chipset. I couldn’t find an option to set the driver for the USB adapter. The only place I found the ability to select the NIC type was in the default bridge (vnet0?). Probably missing something important here.

Update: so I bought a new SSD and installed pfSence onto bare metal and … it works perfectly. No issues at all, video streaming working fine in both directions, CPU usage doesn’t move off 1%, Memory doesn’t move off 4%, zero packet loss, RTT 0.6ms, RTTsd 0.1ms. Even maxing out my WAN at 100Mbs, ping times only increase by 10ms.

So this is without any doubt whatsoever, a Proxmox issue. Maybe a driver issue with the external USB Ethernet adapter?

Can you install device drivers in Linux or are they all built into the kernel?

1 Like
9

PFsense is BSD based, so you need the BSD driver. Here is an old post that will give you the general idea of fetching and installing the drivers, you just need to get the correct one for the pfsense that you have installed (BSD version that was installed).

https://www.reddit.com/r/PFSENSE/comments/t872mx/fix_issues_with_realtek_nic_on_pfsense_260/

10

Greg, I’m thinking here that pfSense is working just fine, the problem appears to be with Proxmox, which I believe is based on Debian.

11

My Realtek NIC is physically on the motherboard; I’d missed that yours is USB. I suspect that you’re not going to overcome the Proxmox/pfSense interface problems with USB.

Did you try simply passing the USB through to the VM & ignoring Proxmox trying to use it completely? That should have a better chance of working.

Of course, all that said, the standard advice is not to virtualise pfSense at all.

12

Most certainly did, tried with and without IOMMU enabled, with and without passthrough, these didn’t appear to have any effect, although with passthrough enabled, the device did not even appear in Proxmox, which obviously meant that Proxmox couldn’t then access the Internet.

I also tried swapping them over so that the USB adapter served the LAN (via the default bridge Proxmox sets up), leaving the on-board NIC to service the WAN … in this configuration, the very moment I started pfSense I lost all connectivity to Proxmox and, on reboot, noted a load of kernel errors basically indicating the kernel driver for the USB adapter had blown up.

This is why I’m currently thinking kernel driver support for the USB adapter in Debian (Proxmox’s OS) might be the root cause.

Lesson learned: never buy a NUC or mini-PC with only one Ethernet port!!

13

OK, I think you’ve conclusively proved that Proxmox doesn’t like your USB NIC :laughing:

FWIW, I’m currently using this Minisforum UM690Pro

It’s really fast & has 2 x 2.5Gb onboard NICs. I’m running the latest Proxmox & a bunch of VMs including pfSense and a Linux-based email server and a few Windows server VMs. Uses virtually no power compared to my old Dell server (plus it’s silent!) and I’ve had zero problems with anything I’ve thrown at it so far.

14

Ahh, man, that looks fantastic, and less than I paid for my Intel NUC 11. I figured that, whilst I know that they’re out of the NUC market now, I’d have fewer problems if I went with Intel!! Definitely a reason to go back in time and slap myself in the face!

15

HP T740 if you can get it cheap, AMD V1756B processor with 16 gb of ram and an SSD of some size and connection (it has m.2 SATA and NVME and I think has a SATA connector for 2.5 inch). Room for a short 4 port card of your choice plus an onboard Realtek gigabit that you can set up for management connection. I did upgrade ram and ssd after purchase, and added an Intel i350 quad port card.

There is a gotcha though, look up atkbd0 [GIANT-LOCKED] and you’ll find the workaround.

I did try the thinOS that shipped with mine, it was fast and I’m thinking of using it for a second machine on my network at work. Remmina and web browser is all I really need for this second machine. Need to fool with this later this summer and see if it will do what I need.