XCP-NG Mellanox


#1

Hello Everyone,

Wanted to get some help in regards to Mellanox and XCP-NG.

Background: Dell R710 (google appliance) , XCP-NG 7.6, Mellanox Connectx-2 PCI-Epress x 8 10GBe, Ubiquiti SFP+ modules LC, Mikrotik 10G switch.

Few days ago my 10G cards arrived for my freenas and xcp-ng. I installed 1 into freenas setup the NIC and all is right in the world. Installed other card into XCP-NG booted it up and opened XCP Center. Check the NIC’s out and it says Disconnected.

Now i know the card is good because I tested it in a windows 10 machine connected to the mikrotik switch, grabbed an ip and everything. So cables good mods are good and switch is good. So the only thing left i can think of is firmware on the card or drivers in xcp.

Yes I know that i have 2 devices right now and that DAC cable would be easier and cheaper but the plan is to add other servers with connection to freenas so we need the switch right now.


#2

Do you see the Mellanox pre-boot during POST on the R710?

Try swapping the cards around. See if the same issue exists. If so, I would likely point to XCP issue.

I don’t use XCP, but I do use a R710 with Mellanox ConnectX-2 Cards, in vmware.


#3

Just rebooted and confirmed the card is definitely visible to the dell (pic 1) and also started getting this error when booting. It will skip on after a while but probably about 10min. (pic2)


#4


#5

I get that same first error message every time I boot my XCP-NG server and it is running on an Intel NUC so it may not be related to your problem.


#6

yeah im not too worried about the error since it always boots and from what ive seen online its not the biggest problem.


#7

holy crap I woke up…responded here and opened xcp-ng center checked the nics and it says Connected…loving the extra zero =]


#8

I’ve come across something similar after rebuilding a host with XenServer, sometimes the ports don’t detect anything connected for a while. Usually we end up fartin around with disabling LACP on the switch, and re-enabling it, and to be honest, I haven’t yet found any one thing that seems to make them detect as connected. Had this issue on Intel cards, as well as Broadcom, although far less often on the Broadcom cards. The good news is once I can get them to show connected, I’ve never had an issue ever, once they are running in production.

So I would say its likely something on the Xen side of things rather than your NICs themselves.


#9

I’m curious sysman923, how do you like your former Google Appliance? I’m eying up an R720 that used to be a Google Search Appliance. Price seems reasonable, has WAY more RAM than I will likely every need, dual 6 core xeon processors. How hard was it to flash the standard firmware on it?


#10

Love it so far, has given me nothing but relentless power and ability to do what ever i want to create in my lab. I would definitely buy another


#11

working great now cant complain.


#12

Good to hear! How hard was it to flash to the standard (non-google) firmware so you could install XCP-NG on it?


#13

I didn’t need to flash it at all actually still has the google bios and idrac firmware. Works fine unless that was the root of some of my issues above.


#14

Interesting, I was under the understanding that you’d have to flash the stock Dell firmware for that server version to get off of the “appliance” mode thing. Weird…


#15

not in my 1 and only experience with a google appliance


#16

Very very cool, I’ll have to do the “forgiveness rather than permission” purchase sooner than latter, especially if they don’t need to be flashed.

What should I expect with installing XCP-NG? I plan to virtualize pfSense, and then setup a couple other VM’s on it. Also, how many (and what type) NIC’s did yours come with? I’ve seen some with 4 built in.


#17

I don’t usually like to virtualize my router in my lab so i use a ubiquiti edge router to do it. Mainly because if i need to update xcpng and it needs internet access wellllll the vm will be off and if i want to switch hypervisors I have to back up the router blah blah reinstall.

I’d rather not lead you down the wrong path since i’m not 100% sure on the whole config. I have done untangle inside of xcpng but not as my main router. It has always been ubiquiti -> xcpng -> xcp virtual switch (for all my random vm’s) -> WAN of untangle -> then attach vm’s I want behind untangle to the PRIVATE lan on the 2nd nic of untangle. I really hope that made sense it made sense in my head.

Anyway my server came with Broadcom 5709 NetXtreme II and I popped in the Melanox Connect X 10G

https://www1.la.dell.com/ni/en/corp/Networking-Solutions/nic-broadcom-bcm5709/pd.aspx?refid=nic-broadcom-bcm5709&s=corp

pic1


#18

This might be of some interest to you…if mine wasn’t setup id probably do this or at least try it =]