Just installed XCP-NG 8.1, upgrading from a single server UnRAID setup to a FreeNAS SAN and separate VM host.
Picked up a couple of Mellanox IB cards, figuring I could make use of a 40GB channel between the VM host and the NAS.
I can’t for the life of me get XCP-NG to activate the card. lspci shows the card is there and it’s loading the mlx4_core kernel driver.
I’ve tried all manner of things from all over the internet, but I can’t seem to get this card to come up.
NOTE: the card is not connected to anything at the moment. I am waiting for some additional hardware to arrive before I move teh UNRAID box to freeNAS. But I would expect it to show up on a ifconfig-a, even if the card wasn’t doing anything yet.
The cards are MT25408A0-FCC-QI. This is the spec sheet I get from Mellanox’s web page:
Please advise. While my hair has gotten rather long because all of the salons are closed, I am starting to remove it in clumps.
I don’t know that those cards are compatible with Xenserver. They have an HCL list here:
http://hcl.xenserver.org/networkadapters/?networkadaptersupport__version=16&vendor=52
Looks like they were in 7.6…
http://hcl.xenserver.org/networkadapters/?networkadaptersupport__version=14&vendor=52
So, is 8.1 that much better than 7.6? Would running 7.6 be a real option?
This is for a home lab setup, but also as a prototype of what I plan to build for my work server arrangement. So I’d rather be proficient in 8.1, but I can’t afford cards that fast that are compatible with 8.1.
So, what’s the collective advice? Go 7.6 and use the existing cards, or go 8.1 and find another solution to get me high bandwith for iSCSI?
10G cards that are comparable are also very affordable and unless you are attaching to an NVME storage array they are plenty fast enough.
Well, I’ve decided to use ESXi Community for now. These cards are still on the HCL, and for my home lab I don’t need any more features than the free ESXi license gives me. Also, until I get fully converted from UnRAID, I am going to virtualize UnRAID for it’s easy Docker interface. Been looking at the others, and they all sem to require more knowledge of Docker than just a point and click download like UnRAID does. Might even keep it as my Docker Host long-term.
Sadly, virtualizing UnRAID in Xen is a real problem. Read lots of threads and tried it and could not get it to work at all. 
I really like using open source software, but since I’m already invested in these cards, I am going to use ESXi for virtualization and FreeNAS for my SAN storage over iSCSI.
Thank you, Tom for answering my questions quickly. Love the YT channel, and maybe one day I’ll get into XCP-NG. I’m sure I’ll be back to pick your brain on FreeNAS as well.
1 Like