Multiple XCP-ng connections to TrueNAS

Computer Hardware & Server Infrastructure Builds
1

Hello all!

After seeing that ESXi free was going away, I dived into the XCP-ng videos Tom has posted. I’ve been really liking XCP-ng so far coming from many years of ESXi but I’ve ran into an issue with it connecting to a TrueNAS server with iSCSI shares.

I have been testing two servers that will be hosting VMs directly connected to this single TrueNAS box each with two 40G QSPF+ DAC cables with XO from sources. The four overall links are in their own 192.168.251-254.X/24 subnet plus the normal network management links.

The issue I’m having is, while in an XCP-ng pool, the XCP-ng servers seem connect to the iSCSI via the master server’s IPs of 192.168.251.2 (or the second link 192.168.253.2) to the TrueNAS 192.168.251.1 (192.168.253.1). The second server (192.168.252.2 / 192.168.254.2) cannot connect at all to iSCSI, even when it’s the selected server in the add storage page, getting the error SR_BACKEND_FAILURE_141(, Unable to connect to ISCSI service on target, ).

When I do add the SR, the master server will show it’s connected on both its paths with the target 192.168.251.1 (and multihomelist of all four of the TrueNAS IPs), however the second server also shows the same target of 192.168.251.1 which is inaccessible. I’ve tried having one TrueNAS iSCSI zvol share that they could both connect to and a zvol pool for each server with the same errors. The only way each of them will connect to their own shares is if they are not in a XCP-ng pool together. The TrueNAS’s iSCSI share settings are basically the default wizard clickthrough but with the XenCompat boxes checked.

I know networking wise, this makes sense. They are four separate networks so the 251/253 and 252/254 networks can’t talk to each other. I figured in a pool, each of the servers would use their own connections to get to the shares. Is this way of connecting the servers incorrect? If so, anything to point me to show a better setup?

Thank you!

2

All the hosts in the resource pool should have matching network settings so that they can all reach the TrueNAS server over the same ETH port. Are you trying to setup iSCSI Multipath? If so there is an option on the advanced tab of the host to enable Multipathing.

My preference is to use NFS because iSCSI on XCP-ng does not offer thin provisioning and NFS does.

3

Enabled multipath on the XCP-ng pool hosts before trying to connect to the iSCSI.

I missed that iSCSI doesnt do thin, I’ll go watch one of your videos on setting up NFS over iSCSI then.

Thank you!

5

Remember, just because it’s thick provisioned doesn’t mean it’s using the full space on truenas because of compression. Tom has a good video out there somewhere about this.:slightly_smiling_face:

6

Yeah that’s a handy feature for sure! With this setup though, I’d need a storage network to make it all work I finally realized (took me long enough!). I liked the thin that NFS offered but since the VM’s probably wont move between servers anyways, just went back to two pools and iSCSI with MPIO for the extra throughput.