pfSense router and Ubiquiti switches/APs - traffic between vlans fail

Hi!

I have a pfsense router (Netgate 2100) and a couple of Ubiquiti switches and APs. My network looks like this:

I have defined 5 networks in pfsense:

  • Main (no vlan)
  • iot (vlan 88)
  • guest (vlan 22)
  • work (vlan 55)
  • admin (vlan 11)

I have defined all networks in Unifi Controller SW and also wifi networks for the networks with vlans above.
To start with I have defined firewall rules in all networks in pfsense so that anything in any net can talk to something in all other networks similar to this:

My problem
On my windows computer (on network Main) I can ssh to the raspberry pi (on network iot). Now, if I from my raspberry pi try to ssh to my synology NAS (on network Main) it fails and the connection timeout.

I have tried to google and my best guess is that I have Asymmetric Routing. I found troubleshooting guide in Troubleshooting — Troubleshooting Asymmetric Routing | pfSense Documentation. I have tried the Automatic Fix which did not fix the problem.

Does anyone have any idea of how I can troubleshoot this to find a solution?

BR,
Niklas

You need a rule to go from the IOT network to the main network. THOUGH, don’t do the whole thing as that would defeat the purpose. Do those IP by IP as you need them, or create aliases to embed a group of IPs into.

Now I feel a bit stupid. Isn’t that the rule I showed above?

What i would do in your shoes is ensure the rules are the same on the MAIN and IOT.
Then test the connection between the PC on MAIN with the Pi on IOT, then move the PC to the IOT and the PI to the MAIN and test again. I would expect the same outcome.

If the above is true but fails for the NAS, then something on the NAS is blocking the SSH connection, perhaps a firewall. Put the NAS on MAIN and see if you can SSH into from your PC.

Ok, I did some testing now.

  1. PC on MAIN, rpi on IOT. ssh PC->rpi is ok
  2. PC on IOT, RPI on MAIN. ssh PC->rpi is ok
  3. PC on IOT. ssh PC-> NAS fail
  4. PC on MAIN. ssh PC->NAS is ok

The NAS is on MAIN seems logical that both rpi and PC on MAIN can connect to NAS.

The firewall on my Synology NAS is disabled.

That to me sounds like ssh across vlans works, I’d guess it’s the synology and perhaps some further configuration on that. Some feature might need to be enabled on it to work in your scenario.

It my be. I guess I don’t know if IOT Subnets is an alias and what’s in it.

I have defined a couple of networks after looking at https://www.youtube.com/watch?v=b2w1Ywt081o&list=PLjGQNuuUzvmsuXCoj6g6vm1N-ZeLJso6o&index=2 by Lawrence Systems.
The following vlans are defined:


The following networks are defined:

The network called MAIN above is named LAN in my pfsense.
IOT is no alias.

You might very well be correct here but I have no clue of what it can be. Is there something that I can test?

What VLAN is the Synology on?? Also, how are your switches configured? What VLANs are they passing?

You also have the default LAN to Any Rule still enabled I suppose? Or else MAIN couldn’t get to IOT.

Looks like everything is OK on the Firewall side, then. Are you SSHing by IP or host name?

The NAS is on MAIN/LAN which has no VLAN in pfsense… It is the default network which I have left as it is. The youtube film I watched above I understand that I should leave it as is.
Should I move the NAS to a network with VLAN?

All switches passes all VLANs. There are a few ports that has only IOT VLAN, e.g. the port where the RPI is connected.

Yes, the default LAN has the any rule.

I’m SSHing by IP.

Generally not a good security practice to send all Vlans unless it is needed…for example a trunk port. If a device cannot accept more than 1 Vlan then I don’t know if the communication will be reliable. Try sending only needed Vlan to the clients and block others. Keep trunk ports between switches, Pfsense, access points, and the network controller. See if that helps

Generally not a good security practice to send all Vlans unless it is needed…for example a trunk port.

I agree with this and my intention is to restrict this but first I want to solve the problems I have. I don’t think it will be easier to solve by adding more limitations.

I will try to follow your proposal.

If you are plugging your NAS into a trunk port, I doubt that will work for you. If you have more than one NIC, unplug all except one and test again.

1 Like

You wrote

Keep trunk ports between switches, Pfsense, access points, and the network controller.

I have my Unifi network controller currently on my Windows PC which is not connected to a trunk port. Did you really mean to include the network controller in the list above?

I have now made several changes based on what @neogrid, @jeff3820 and @chrisfonte have written above:

  • Only pfsense, switches and APs are connected to the trunk ports.
  • I realized that the Default network in Unifi network controller in on 192.168.1.0/24 which I did not have as default network in pfsense (and I did not understand how to change it in the network controller). Now I have 192.168.1.0/24 as default network in pfsense also and all unifi switches and APs are on this network (and nothing else).
    This change was tricky to do so it took me some time.

My network definition in pfsense looks like this now:


LAN is the default network as 192.168.1.0/24.

I now have my windows PC and NAS in network SKYNET and I have the rpi in network IOT.

  1. I can ssh from the PC in SKYNET to the NAS in SKYNET (the NAS has ssh enabled)
  2. I can ssh from the PC in SKYNET to the rpi in IOT.
  3. On the rpi I can’t ssh to the NAS in SKYNET
    I don’t understand why this is a problem.

As I said earlier I currently have firewall rules that as I see it enable traffic from every network to all other networks. I will change this later but for now that is how it is defined. For IOT I have this single firewall rule:

How do I troubleshoot this?

Does this Rule you’re showing under IOT exist on all other Interfaces as well?

Would be good to see firewall rules on LAN, IOT, Guesty, Work, and Skynet networks