Can't make LAN interface routing work

Hello, Tom’s world. Please help.

I have a Dell R630 colocated in a datacenter. The server has 2 x SFP+ and 2 x 1GbE RJ45 ports. 10GbE ports are connected to 2 switches in LACP mode. The 1GbE RJ45 ports are hanging in the air.

I run pFsense in a virtual machine on top of XCP-NG. Among other things I have a TrueNAS virtual machine with PCIe HBA passed through to the VM.

I need to provide access for the host-level OS (hypervisor) to the TrueNAS VM so that VM’s virtual disks can reside on an NFS share and VMs can get backed up to an SMB share. Not easy to get it done since only public IP address is assigned to the bond. So, I figured that I will assign a private IP address to the bond made of the 1GbE interfaces by issuing command
xe pif-reconfigure-ip uuid=558988b2-e595-ab95-4e46-12beaaca40c3 netmask=255.255.255.0 gateway=192.168.30.254 IP=192.168.30.1 mode=static and then add this bond as an interface LAN2 in pFsense. If this worked then XCP-NG would be able to talk to TrueNAS via this 1GbE interface.

This resulted in the following configuration:

pFsense WAN: Public IP address
pFsense LAN1: 192.168.20.254
pFsense LAN2: 192.168.30.254

XCP-NG host eth2 192.168.30.1

I added default allow to any rules for IPv4 and IPv6 on LAN2 which should have enabled routing across LAN interfaces as the same rules come by default on LAN1 with a fresh install.

Problems

  1. I can ping XCP-NG host interface from pFsense ping window (192.168.30.1). I can also ping LAN1 from LAN2 and vice versa.
  2. I can’t ping LAN2 on pFsense from the host despite those ‘allow to any’ rules were added.
  3. If I add an explicit rule to allow 192.168.30.1 to LAN2 then I can ping pFsense LAN2 interface but nothing goes beyond it - LAN1 and any other IP address on the virtual machines don’t get ping. So there is communication between eth2 on the host and LAN2 on pFsense but nothing routes past LAN2. Adding explicit rule where source is eth2 on the host (192.168.30.1) to LAN1 in pFsense does not help.

Am I missing anything in this setup? Why LAN-to-LAN routing is not working? Seemingly simple matter has eaten 2 days of my time. Please help with some advice, ideas and suggestions.

I’d take a look at the routes / routing table in pfsense.

No much there to see, really

and on the xcp host. and if you ping a machine in LAN1, also the route table of that machine.

did you look (tcpdump / wireshark) at the ICMP request datagrams when they enter LAN2 interface and when they leave LAN1 interface? the same for the ICMP response datagrams when entering LAN1 interface and when leaving LAN2 interface?

The reason for doing this exercise is to determine exactly where in the path the datagrams get lost.

I did not do tcpdump yet.
This is what I can get from the host now

xenbr2 is the bond interface?

always good to use “-n” in the command to only have IP addresses, no host names.

I cannot know what “gateway” is, if this is a upstream machine, or the pfsense VM.

look at tracepath -n 192.168.30.254 and at tracepath -n 192.168.20.254

The second line on the screenshot is the bond consisting of 2 x 10GbE interfaces. This is also what caught my eye - the bond is on xapi, while the interface in question is on xen bridge.

the first line is default gateway 0.0.0.0 …
and gateway here seems to be a host name, to avoid this use route -n or netstat -rn

I am sorry, I now have re-read your initial description for 4 or 5 times and it doesnt get clear. I think I dont understand the actual setup suficiently.

Also you haven’t done the tracepath.

I cannot help you, sorry.

I had to step away, apologies.

Here is route one more time

With tracepath things get clearer slightly

when running tracepath -n 192.168.30.254 I immediately start getting no replies, which makes me believe the tracepath request goes through the 10Gb bond, i.e. the host interface with the public IP address.

When running tracepath toward a a host with public IP address I get proper tracing.