Extremely wierd multiple NIC problem using pfsense, unifi and trueNAS core, scale and proxmox

[formatall]

So I don’t know where to begin and why such a weird behavior or last may be I am not understanding networking…

I tried my best explaining and what I did so far. I didn’t assign default gateway on both NIC, I only did on one.

So I have pfsense and unifi switch set up this way:

This is problem on 10.0.10.110 only. Happens with TrueNAS Core, Scale and Proxmox. Set up is almost same as of all system has two nic and all of them gives hard time on 10.0.10.1/24 network.

I tried using different NIC (Separate PCI slot and Motherboard NIC) on server side and same behavior. I tried changing nic on pfsense to IoT network (NIC - 6) just incase if NIC-2 is bad, but same behavior.

iperf test between two systems - as you can see 10.0.10.110/24 - BAD, but 10.0.60.110 - GOOD. I did same test between Raspberry PI which connected on PORT 10 of switch with Default profile, I am getting full 1 gig connection and it doesn’t drop as it does with multiple NIC on three system I have.

Unifi Network List:

Server60 profile settings:

ServerUPlink profile settings in unifi:

I would really appreciate your help as you do this set up all day. If you want me to clarify something or want more screenshot, please feel free to ask.

[formatall]

I did one more thing to see if I disconnect my dac cable which is on 10.0.60.1/24 network, and I can only use 10.0.10.1/24 and problem goes away.

Guess what, I can’t even access 10.0.10.110 at all. At least I was able to access but it was keep disconnecting when both cable were in use.

How is it possible if my both connection are two physical cable on two different subnet.

[Paul]

Yes, you can have different networks ports on different subnets on the same machine - only one should have a default gateway. The other settings, leave gateway as blank

The default gateway is added to the network port that you want internet access on

[Paul]

are you using one cable from the pfsense to Unifi Switch ?

If so, you should really assign a vlan to 10.0.20.0/24 network and make the necessary Unfi changes.

Explain in more details the server_uplink and what is attached to port 5 on the switch.

[formatall]

I had that from beginning, but still not working. How come it is not working when I disconnect SERVER60 cable then whole connection break, but not other way around. And even though both cable are connected, one gives problem. One of the port and why does there is no issue on 10.0.60.0/24 network but on 10.0.10.10/24?

My third port on pfsense was not in use, so I decided to create new network 10.0.20.0/24 and try to connect that to truenas scale and same issue. I thought ALL profile on 1st port was causing that, but it happens on port 2 - coming from port 3 in pfsense.

[formatall]

on port 5, it is my fifth physical interface from pfsense which has ip range of 10.0.60.0/24. That ip range isn’t giving any issue.

Here is screenshot of my pfsense interface:

Beside, I physically attached igb2 directly to one of NIC on truenas scale and kept other to same sfp+ port coming from unifi switch and tried eliminating any switch component to see if it happens on second NIC. Guess what? It keep happening. While 10.0.60.110 - no issue so ever.

[igb2 - 10.0.20.0/24 - Testing on separate interfaces as I can’t detach 10.0.10.0/24 as my other devices attached to that network works fine - they all connected through unifi switch]

so I don’t think it has anything to do with switch anymore but to do with pfsense or at least I think. I hope @LTS_Tom look at this and see if he experienced something like this.

[LTS_Tom]

Not an issue I have encountered.

[formatall]

Any suggestion where to look for?

Because, I used different NIC ports on TrueNAS Scale, Created completely new interfaces on pfsense and used same setting as 10.0.60.0/24 network (Works fine) but different subnet on that and see if connecting to two will cause, which it still an issue, I used completely different switches for those two interfaces (One going to toward unifi and other going toward tp-link unmanaged) which are going to TrueNAS scale, I even removed switch all together from second NIC and attached directly to pfsense and issue still persist.

Ping doesn’t drop at all, just can’t do iperf3 test or UI keep refreshing every few seconds (with proxmox, console keep getting disconnecting and no syslog change) and only happens on second NIC and not my 10gig network. And it happens with Core and Proxmox. If I remove SFP+ cable. from truenas scale, I can’t even open UI if I type 10.0.10.110 or 10.0.20.120 (depending where I am connecting cable) - keep disconnecting is secondary issue at that point.

I am completely lost and I don’t know what else to look into to find issue.

[pavlos]

so the problem is with NIC2 (10.0.10.0/24).
Is DHCP Server configured correctly for LAN?
Does Status > DHCP leases show the truenas?

[formatall]

It is only problem when I have two nic connected to same server. Other subnet works fine.

DHCP server is configured fine on LAN - 10.0.10.0/24 - as I have three raspberry pi, two unifi devices - they all work as it should be and they don’t get disconnected or no problem.

yes, DHCP leases does show truenas. In fact, I can even open truenas scale, core, and proxmox UI using that IP assigned to second NIC to truenas, but it keeps getting disconnected until I refresh page or in proxmox case, I click on different tab and come back to console tab. But when I connect to any of those UI using 10.0.60.0/24 - my 10gig connected devices using sfp+ in unifi and unifi connected to pfsense, it has no issue at all. Initially, I though it has to do with switch configuration, but then I connected second NIC directly to pfsense second interfaces under different subnet (10.0.20.0/24),[ because my other devices works fine under 10.0.10.0/24 and didn’t wanted to create interference] , and it still has same issue even thought it is directly connected to pfsense device.

If I remove 10.0.60.0/24 cable from truenas, I can’t access UI from other IP as well. Curious is why? Because I am not removing that cable. I understand that disconnect may happen, but I can’t even open UI. That’s weird behavior as well.

Does pfsense has setting which I missing which use one cable to transfer both subnet traffic through. I know this is stupid question, but hey that’s whats happening if I remove my sfp+ cable, I can’t even access UI from ethernet 1 gig connection.

[pavlos]

According to truenas, only one interface can be DHCP. When I tried to configure my truenas to use both NICs as DHCP, it complained. I set one as static and all is well. Could that be your issue?

[formatall]

so my 10gig is static and other is DHCP. In fact, I tried both static as well and same issue.

[formatall]

@LTS_Tom

Any suggestion where look for?

Because both physical interfaces works fine when I use one at a time and disabling other one, but when I bring up both at same time, that’s when it doesn’t work.

I did only use 1 gateway address and both of them are on separate subnet. One is on 10.0.60.110/24 and other is 10.0.10.110/24.

As I said, if I have both of them UP, removing cable from 10.0.60.110, I can’t access UI from 10.0.10.110 network, but removing cable from 10.0.10.110 - there is no effect on 10.0.60.110 and I can access UI from 10.0.60.110 without issue.

ADDRESS IS DIFFERENT IN PICTURE, BECAUSE I AM TESTING DHCP AND STATIC TO SEE EFFECT. BUT IT IS SAME BEHAVIOR.

and when both of them are UP, 10.0.10.110 - keep disconnecting and reconnecting.

[pavlos]

Not sure if this helps but I can access the truenas ui only from the network that was configured dhcp.

My truenas is 10.0.0.22 dhcp and 10.0.20.2 static but I can only access the ui from the first ip.

[formatall]

You see, you should be able to access UI from both IP. I can do that but it keep disconnecting.
Does your second IP connected only for storage need and directly attached to that computer or can you access second IP from any PC.

[pavlos]

I have two networks separated by a pfsense firewall, the primary 10.0.0.0/24, the secondary after pfsense is 10.0.20.0/24 I can only access the ui from the primary network. My ascii diagram …

primary 10.0.0.0/24 ---+---------- clients
                       +
                       + truenas (igb0=10.0.0.22, igb1=10.0.20.2)
                       +
                       + secondary 10.0.20.0/24 ------ clients