XCP-ng + WinServ19 + XenServer PV network drivers 'resetting'

Hello,

This is one of those things that I wondered if it would be better to consult with Vates support since we sub to both XO and XCP-ng support, but this is also within the realms of Microsoft and Citrix, so maybe ask where I know at least some folks here have some experience with all of this at scale.

This issue is with the Citrix XenServer PV network drivers on Windows Server 2019 guests within XCP-ng+XOA. Server 2012 R2 also had the problem, but those have been migrated to 2019. This also does not happen on all 2019 guests, but there is not an apparent pattern for whether a DC or another type of server. Each has ‘Manage Citrix PV drivers via Windows Update’ enabled.

After some time, an interface on one of these servers will ‘reset’. The interface name may change to some iteration of ‘Ethernet X’, and most commonly the network type will change from Domain to Private. The quickest fix I have is to uninstall the device in device manager, without removing drivers, and restart. After restart, replace static IP info, and restart DHCP server, if needed.

I have not seen any pattern to this yet, or underlying cause. Nothing sticks out in Event Viewer, but I am not sure what to look for.

Has anyone seen anything like this?

Do you also have the xcpng version of the PV drivers installed or only the citrix agent?

Citrix drivers from Windows Update. Citrix agent 9.3.1-91 from Citrix.

The Citrix agent normally installs Citrix drivers and competes with the windows update setting in XOA, one or the other is suggested.

You can try unchecking the windows update in XOA and just install the full Citrix set, but you might need to manually remove the drivers from the update. I’ve had the same thing happen when both the update and Citrix were being done. Thankfully this was on my lab system and I just destroyed the VM and started over.

Kinda. Default is to install the Citrix PV drivers, but to disallow driver updates by the agent. Since I am needing just the management agent, I /should/ be able to just uncheck ‘Install I/O Drivers Now’.

image

Besides, that does not necessarily explain why the network type changes from Domain to Private, which is at the crux of the entire problem. This changes the local firewall profile, breaking DHCP, or whatever other services would only be enabled in the Domain and not Private profile.

Also, from VM | XCP-ng documentation

" If you are using Xen Orchestra, you can switch the “Windows Update tools” advanced parameter on from the “Advanced” tab of the VM view. This will install the device drivers automatically at next reboot :warning: but not the management agent which still needs to be installed from Citrix tools’ installer.

… So the “Windows Update tools” option is not a complete solution if you need the guest metrics from the management agent. However it may be a convenient way to get future driver updates if you wish so."

I think this issue is similar with this issue:

Oh I have no idea how I missed that. Seems super promising! I will give it a try tomorrow and try to remember to report back. Thanks!

While it would be nice to know the root cause, sometimes it’s nice to have an easy fix that works and doesn’t add much to workflow.

What you think and what actually happens may be two different things. Reading through the linked thread, you are just safer using the Citrix drivers while you are installing the management agent. Would be nice to have XCP drivers and agent, but that seems to how been put to the back burner.

I’ve found that my systems are most stable with the Citrix stuff, shame that it is so hard to get now.

Greg, I really do mean you no offense, but did you read? At no point did I say I am using the XCP-ng guest tools. I am using the Citrix drivers from Windows Update, and the Citrix agent from their msi. Using the drivers from the msi would be nice, but these are production guests and driver updates through Windows Update are preferred.

This is exactly what Tom Lawrence talks about here:

I would ultimately prefer using the XCP-ng tools and drivers, but since these guests are Windows, I will wait until that project is more mature and not done basically by one overworked developer.

No, I read it. And not offended at all. If you are installing the management agent, you might as well let that same tool install the drivers. It just requires some extra steps. If you want I will detail exactly what I’ve worked out to save myself some hassle.

And again, I only mentioned my workflow because that’s what I found to be stable. I also found my system to be unstable when I mixed and matched and tried to install over the top… It just got to be such a mess that I destroyed the VMs because it was faster to start again than to undo all the garbage I caused. Thankfully this was on my lab system, not my production system. One other issue on my lab system is that I’m using old Dell C1100 with Xeon x5650 processors, this does not really help, my production system seems to work better since it has modern hardware. Hoping to beg some old servers from a business that will at least be in the Xeon E5 v2 era.

The problem with the network dropping out is probably a different driver “creating” a new NIC in Windows. Again, I’ve had that same issue and it stopped when I just started using the Citrix tools in their complete package. And I may be wrong, but I thought Citrix had a note about this saying that drivers can flip back and forth and cause the network to change, and that you need to pick one way and go forward.

I should also state that most of what I’ve been doing is either Server 2019 or Server 2022, and moving everything to 2022 once we get out of the semester and I can move the last physical AD and do an in-place upgrade on one of my member servers, this member isn’t doing much yet so not concerned about an in-place upgrade. It mostly runs other tools like the NDItools suite and Windows Admin Center, and it’s mostly a placeholder because I need to keep at least 5 servers running to keep KMS working (unless they changed the requirements).

I will say, I’m looking forward to when we can have XCP-NG drivers and agent working, but I do not have the skills to help make this happen. Citrix does not make it easy to get their tools.

Hope this helps anyone trying to get a more recent version of the agent

http://downloadns.citrix.com.edgesuite.net/17461/managementagentx64.msi

1 Like

When creating a new VM with a VIF, do not boot immediately after creation and enable ‘Manage Citrix PV drivers via Windows Update’, upon first boot that VIF will be using an Intel driver. If you go straight to Windows Update, the entire Citrix PV driver set will be installed. The (hidden) device with the Intel driver can be uninstalled on a later boot. This would be the better option for production use as those drivers are automatically updated through Windows Update. I would trust that to manage my drivers far more than a management agent that seems to be obfuscated more and more by Citrix.