Xcp-ng x520 es25519

Computer Hardware & Server Infrastructure Builds
1

So I have an old Dell C6320. Lovely system, runs 4 nodes in 2 U, basically a bunch of fancy R630s. Love the thing. Runs proxmox, debian, and anything else I throw at it like a champ.

I decided to try running XCP-NG in the thing and see if I can get some extra use out of it. Well, on the first node I fire up, I’m hitting a solid 30-50% packet loss on the internal X520 NICs. 82599ES chipset. Onboard.

First I think there’s something wrong with the hardware and I go to research how much it would cost to replace a node. Too much for today. I switch to the Mellanox NICs in the Mezzanine slot, and keep going.

Well, I decide to get to the bottom of this, I create a simple debian 13 VM, and connect it to the Mellanox network bridge, and ping my internet gateway. Works fine. No packet loss.

Then I attach it to EITHER of the two X520 NICs and I get heavy packet loss, 30-50%. This is odd. But it gets odder.

To verify that it’s not a hardware problem, I configure the X520 NICs for PCI-E passthrough, and pass them through to the debian guest. I bring them up in the guest, and they work just fine. 0% packet loss.

Pass them back to the host, go through all the reboots, and then I get back the high packet loss rate.

In the dmesg output on the host, I see the following block every time packet loss happens.

[ 1507.783650] ixgbe 0000:81:00.1: removed PHC on eth1
[ 1507.855479] ixgbe 0000:81:00.1: Multiqueue Enabled: Rx Queue count = 8, Tx Queue count = 8 XDP Queue count = 0
[ 1507.892829] ixgbe 0000:81:00.1: registered PHC device on eth1
[ 1507.955588] ixgbe 0000:81:00.1 eth1: detected SFP+: 4
[ 1509.896477] ixgbe 0000:81:00.0: removed PHC on eth0
[ 1509.971228] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 8, Tx Queue count = 8 XDP Queue count = 0
[ 1510.007199] ixgbe 0000:81:00.0: registered PHC device on eth0
[ 1510.072358] ixgbe 0000:81:00.0 eth0: detected SFP+: 3
[ 1511.435339] ixgbe 0000:81:00.1 eth1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 1513.451352] ixgbe 0000:81:00.0 eth0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

Based on some reading, I’ve attempted both of these flags on the boot cli in an attempt to resolve the issue. pcie_aspm=off ixgbe.allow_unsupported_sfp=1,1 Absolutely no joy. Still getting high packet loss. Similarly, I tried installing the alternate drivers intel-ixgbe-alt. I’m near my witts end, and stumped as to what to try next. Why does XCP-NG hate my X520 NICs?

I do have some actual R630 systems I can test with, all with X520 NICs, but they’re in various state of missing CPUs or RAM or HDDs, and I’m currently recovering from a leg injury which means I can’t physically access them to get them into working order. If this is still unresolved in a few weeks when I can walk right, I’d be willing to test on a regular R630.

2

I don’t have any of those to test I would also post in the Home | XCP-ng and XO forum

3

I have a pile of PCIe x520 cards, and never had an issue, all of mine are Supermicro variant though.

Have you checked to make sure all the firmware is up to date? I’m guessing the generic Intel firmware may not work on those internal chipsets and you’ll need Dell specific firmware, but I could be wrong.

I would also suggest posting on the XCP forums, they may be able to help sort out the issue.

4

I’ve used Dell DRM to create an up-to-date firmware update ISO as of about 6 months ago, and there haven’t been any major firmware updates since AFAIK. Also given the age of the hardware in general and the chipset, I wouldn’t expect them to introduce breaking firmware changes that late in the product lifecycle and not patch them right away, so :woman_shrugging:

5

The only other thing I can think of is hardware offloading vs no hardware offload. Have you tried offloading to off?

6

No joy. I just signed up for the xcp-ng forums. I’ll copy this post over there and see what they have to say about it, probably tomorrow, when they get around to approving the registration.