10GB Direct Connect XEN/FreeNas

I am trying to make a direct connection between Freenas and Xen server. I have Chelsio N320E dual nics in each machine. Configs are as follows:

mtu 9000

Xen Server
mtu 9000

I am not getting any connectivity, please see the picture

I have 4 DAC and i have tried different cable same result. Can someone let me know what I am doing wrong?


Your static IPs are not on the same network.

According to your screenshot because what you wrote in post is different.

192.168.10.x – Freenas
192.168.11.x – Xen

Great video too.

1 Like

Actually, they are. As you may have notice it is a dual port card

port 1 cxgb0 has connected with on xen server
port 2 cxgb1 has connected with on xen server

which corresponds to xen server

port 1
port 2

I used both ports for connectivity incase there was a problem with one of the port.

You are correct, I’m sorry.

Anyone has any other suggestions?

Did you try a reboot? Are you plugging in two cables in both ports in both cards simultaneously? If yes, try one cable and set all ports on the same subnet. Also, try to ping between the machines instead of relying on the UI.

I found when I setup my sfp+ connection between FreeNAS and Xen that once I had the cable plugged in to both machines, I had to reboot FreeNAS and let it come up and then reboot XenServer. After that they were able to communicate.

I am able to ping Freenas NIC from freenas command line, however I am not able to ping ip assinged to xen server ( from xen command line. It does show that it’s disconnected. Even though Freenas shows that there is “no carrier” I am still able to ping. I have ordered a new cable I will see if that makes a difference. Let me know if you can think of anything else.

Yes, I have rebooted both machines multiple times. I have tried single port and multiple ports. Of course corresponding ports being on the same subnet. I have also tried pinging Freenas ip from xen and xen ip from Freenas no response.

FreeNAS output

Xen output

That Chelsio card might not work in Xen. Can you run a command that shows the drivers that have been loaded? Something like dmesg or modinfo in Linux. I am not experienced in Xen.

Instead of buying more DAC’s I would buy an Intel X520 to put in the Xen machine. Intel X520 cards are in the supported list for all Xen releases.

It is using a cxgb driver


@zafars Did you ever figure this out?

I believe the nics were defective, since I recently got em I end up returning the N320E and got Intel X520 which is even compatible with ESXi. I end up using the same DAC cable that I had ordered for those nics and everything worked. Just need to restart the NAS after doing the connection. Thank you everyone for your suggestions and help.

1 Like