Dropping Connection XCP-NG with Free/TrueNAS as guest

Hey guys,

I have updated my FreeNAS to TrueNAS and since then I am not able to connect to the iSCSI block anymore. Every time I try to connect I can see in the console of TrueNAS the following error
WARNING 192.168.10.10 (iqn.2020-02.com.example:XXXXX): no ping reply (NOP-Out) after 5 seconds: dropping connections

I have tried to find something on the web, but some say MTU value (is set to 1500) and this is pretty much all. Jumbo Frames are not enabled so this should case the issue. The only way to get back on the NAS is a restart.

As I am not so deep into networking as I would love to, anyone here with some advise?

XCP has 1 card, which has been set up and the network added to the guest vm is Main. I also tried to add a virtual one to the guest, but since XCP does not have a route to it (unable to ping that IP) I am lost…

Thanks for your help

1 Like

If you can’t ping then you don’t have an iSCSI issue, you have a network issue that is causing iSCSI not to work. You need to post more information about your setup in order to dive into why it’s not working.

1 Like

I can ping the devices as long as I don’t want to add the storage

From XCP to NAS

PING 192.168.10.20 (192.168.10.20) 56(84) bytes of data.
64 bytes from 192.168.10.20: icmp_seq=1 ttl=64 time=0.139 ms
64 bytes from 192.168.10.20: icmp_seq=2 ttl=64 time=0.232 ms
64 bytes from 192.168.10.20: icmp_seq=3 ttl=64 time=0.269 ms
64 bytes from 192.168.10.20: icmp_seq=4 ttl=64 time=0.282 ms

--- 192.168.10.20 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3050ms
rtt min/avg/max/mdev = 0.139/0.230/0.282/0.057 ms

From NAS to XCP

PING 192.168.10.10 (192.168.10.10): 56 data bytes
64 bytes from 192.168.10.10: icmp_seq=0 ttl=64 time=0.152 ms
64 bytes from 192.168.10.10: icmp_seq=1 ttl=64 time=0.289 ms
64 bytes from 192.168.10.10: icmp_seq=2 ttl=64 time=0.350 ms
64 bytes from 192.168.10.10: icmp_seq=3 ttl=64 time=0.204 ms
64 bytes from 192.168.10.10: icmp_seq=4 ttl=64 time=0.226 ms

--- 192.168.10.10 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.152/0.244/0.350/0.069 ms

Setup:
XCP as host with FreeNAS on same SSD. The Main Network is attached to the NAS and it has access to internet and internal to the pihole. The gateway is a UDM which has the VLAN 10 for the XCP, NAS and other server (once the iSCSI is back). Both the XCP and the NAS have static IPs set on the UDM and in the system. The pihole is in the management network with 192.168.99.5 and is setup as the DNS for the VLAN 10, both can ping this device as well.

As soon as I go to create new SR in the XenOrchestra or XCP Center, I add the type to iSCSI and provide the IP, I can select IQN and the LUN, but from this point the connection to the NAS is lost.

PING 192.168.10.20 (192.168.10.20) 56(84) bytes of data.
From 192.168.10.10 icmp_seq=1 Destination Host Unreachable
From 192.168.10.10 icmp_seq=2 Destination Host Unreachable
From 192.168.10.10 icmp_seq=3 Destination Host Unreachable
From 192.168.10.10 icmp_seq=4 Destination Host Unreachable
From 192.168.10.10 icmp_seq=5 Destination Host Unreachable
From 192.168.10.10 icmp_seq=6 Destination Host Unreachable
From 192.168.10.10 icmp_seq=7 Destination Host Unreachable
From 192.168.10.10 icmp_seq=8 Destination Host Unreachable
From 192.168.10.10 icmp_seq=9 Destination Host Unreachable
From 192.168.10.10 icmp_seq=10 Destination Host Unreachable

--- 192.168.10.20 ping statistics ---
11 packets transmitted, 0 received, +10 errors, 100% packet loss, time 10177ms

As I have no idea what further information is needed, any hint into what is needed is appreciated

1 Like

Not sure what the issue is, but I also don’t recommend virtualizing TrueNAS as it can create very strange issues that are beyond my ability to help with.

1 Like

We just finished a building a lab server XCP-NG 8.1 and TrueNAS Core RC1 and dumped it immediately. Something very odd is going on and it is not behaving correctly. We are going to attempt to rebuild with XCP-NG 8.1 and FreeNas 11.3, we do know it works with XCP-NG 8.0 and FreeNAS 11.3.

However per IX Systems requirements for production use we do PCI passthrough for the storage controller to be used by FreeNAS. We are doing this on the supermicro sys-5018d-fn8t, we use the m.2 for xcp-ng install and its local storage for the FreeNAS VM, allowing us to pass the whole SATA controller to FreeNAS.

You may run into and issue with SATA controller passthrough where you have multiple ada0 partitions. If this happens rebuild the FreeNAS with out the controller passthrough. Then go into tunables and add "hint.ada.0.at=“scbus100” power off Freenas. the do the controller passthrough.

Don’t mean to revive this old thread, but did you ever figure out your issue? I have the same setup and just upgraded XCP-ng from 7.6 to 8.2 and then upgraded FreeNAS 11.3 to TrueNas 12 and am seeing the same behavior.

Have XCP-NG 8.2 and Truenas 13.0 as a guest (with LSI Sas2308 passed through) and see same behavior; Truenas network is borked as soon as an iSCSI connection is initiated to it. I’ve tried disabling all hardware offloading for xn1 and it’s vif, but that does not help. Need to reboot vm to re-enalbe xn1.
@sdfungi have you ever been able to fix this?

1 Like

I’ve got the same issue with XCP-NG v8.2.1 and TrueNAS v13.0-U6.1. I came across a post on GitHub stating - TrueNAS guest iSCSI connectivity issue · Issue #538 · xcp-ng/xcp · GitHub

I came here since I have the exact same problem, while testing TrueNAS for future use. (I won’t have it as a VM then, but during testing it is of course an easy route). I can add that the problem seems to be due to page crossing in xenvif_count_requests, in xen-netback/netback.c:

[318644.585621] vif vif-14-2 vif14.2: Cross page boundary, txp->offset: 0, size: 8900
[318644.585635] vif vif-14-2 vif14.2: fatal error; disabling device

Edit: the problem is in FreeBSD 13’s netfront. It is fixed in 14, but TrueNAS core does not seem to go there.
freebsd/freebsd-src@dabb3db

I’ll try TrueNAS Scale and see if there are issues since it’s debian based not FreeBSD.

I would recommend using NFS because iSCSI is thick provisioned and NFS is thin provisioned.