@chrisfonte / @jeff3820
Here are the other rules:
LAN
IOT
GUEST
WORK
SKYNET
There is nothing under “Floating”
IOT
GUEST
WORK
SKYNET
Post the network config for your NAS device.
Plug your devices into that first switch coming off the router then do your testing again.
I’d guess you have a config error in either your switches or synology.
I run a Synology NAS and here is what I guess you want:
General
Network Interface
Traffic Control is empty
Static Route is empty
Connectivity
During troubleshooting I have disconnected LAN 2 for the NAS.
I connected a laptop to first switch after router. The NAS is still connected via another switch. A put the laptop on a few networks and tried to ssh the NAS:
- From Skynet (same as NAS) => OK
- From Work => NOK
- From IOT => NOK
Seems like same result as earlier.
It looks ok to me, I still think that it’s something related to the NAS or the switch but there’s nothing that obviously stands out.
I assume when you access the NAS over https across vlans it fails the same as SSH, if it does then it’s the NAS if it doesn’t then something is up with SSH.
Assume you are using IP addresses instead of hostnames.
Yes, I’m at a loss. Everything on the pfSense side looks fine.
You have pass rules everywhere so interVLAN should pass if the switch ports are setup appropriately. I know you can’t ssh to NAS from IOT, is ping dropped as well?
I am really out of ideas. Again, I use trunks between switches, APs, and controllers…then single networks with block all for all other switch ports.
Have I configured the trunk port wrongly?
Here are the switch configuration for the switch that connects two NASes that I have:
Port 1 is the trunk port
Port 7 is NAS 2, LAN 1 (SKYNET, VLAN=11)
Port 8 is NAS 1, LAN 1 (SKYNET, VLAN=11)
Port 1 is configured like this in Unifi Network Controller:
The Native VLAN / Network popup includes the following choices:
- Default 192.168.1.0/24
- iot (88)
- byvallen (22)
- work (55)
- skynet (11)
- none
Have I configured the trunk port correct?
I don’t have any unifi kit, but I believe they made some changes to how vlans are setup, you should make yourself aware of these changes and inspect your settings.
The trunk is correct
First thing first: before SSHing anything, do your devices ping in both direction? If not, fix that.
2nd: Is pfsense really routing everything or if perhaps you have a switch that do some L3 stuff? Keep all your switch in L2 mode.
3rd: on pfsense, do you see the MAC of both your NAS and Pi ? If not, then frames dont even reach pfsense.
4th: Do you have ACLs on your Synology that would allow only a certain IP or range IPs to connect? Is there a firewall up on that Synology?
5th: can you Main computer ping the Synology ? (they are on the same vlan so they should). If not, your have a network design problem or badly configured switches.
Remember that you are going from VLAN88 to VLAN1 (native).
Also, seeing that you are using Ubiquiti switches, be sure that both PVID (frame arriving on your port or ingress) and Untag VLAN (frames going out of the port or eggress) are the same for that port.
So the port where your Rasberri Pi is should be flagged PVID 88 and untag 88 to be sure the frame remain on the same vlan (entering and exiting). As for your Synology, PVID = 1 and Untag = 1 also (should be by default).
Good luck!
First thing first: before SSHing anything, do your devices ping in both direction? If not, fix that.
I don’t know how to ping from the NAS so I moved one of my rpi:s to Skynet. I’m logged in to an rpi in SKYNET and an rpi in IOT.
From rpi@SKYNET I can ping both my NAS@SKYNET and the rpi@IOT
From rpi@IOT I can ping my NAS@SKYNET and the rpi@IOT
So, check on first.
2nd: Is pfsense really routing everything or if perhaps you have a switch that do some L3 stuff? Keep all your switch in L2 mode.
I only have unifi switches (USW Flex Mini and USW Lite 8 PoE) and have not activly changed the switching to either L3 nor L2. In the settings I can only find ‘layer’ in Settings->System->Advanced and it says:
The information text say ‘Make this UniFi OS Console discoverable on a Layer 2 network’.
Is this the place to configure this or should it be done in another way?
3rd: on pfsense, do you see the MAC of both your NAS and Pi ? If not, then frames dont even reach pfsense.
Yes, I have configured a static IP for both in the DHCP server(in SKYNET for the NAS and in IOT for the rpi).
4th: Do you have ACLs on your Synology that would allow only a certain IP or range IPs to connect? Is there a firewall up on that Synology?
The firewall in the NAS is not enabled.
5th: can you Main computer ping the Synology ? (they are on the same vlan so they should). If not, your have a network design problem or badly configured switches.
Yes
Remember that you are going from VLAN88 to VLAN1 (native).
Also, seeing that you are using Ubiquiti switches, be sure that both PVID (frame arriving on your port or ingress) and Untag VLAN (frames going out of the port or eggress) are the same for that port.
So the port where your Rasberri Pi is should be flagged PVID 88 and untag 88 to be sure the frame remain on the same vlan (entering and exiting). As for your Synology, PVID = 1 and Untag = 1 also (should be by default).
I don’t really understand this part. My NAS is on SKYNET (VLAN11) so what I want is to go from VLAN88 to VLAN11. Or, do I misunderstand you?
The port where the rpi is connected is configured like this:
Today I tried again to ssh from an rpi@IOT to the NAS@SKYNET and today it worked… This has failed for a long time.
SSHing was not the main goal. It was the testing for me. I have a MQTT Mosquitto server running on the NAS the the rpis need to connect to the MQTT server.
Connection from rpi@IOT to MQTT server on NAS@SKYNET still fails. I have a python mqtt client running on the rpi:
pi@housepi2:~/SW $ python3 readSensors.py /mnt/1wire/
Traceback (most recent call last):
File "/home/pi/SW/readSensors.py", line 98, in <module>
client.connect(broker_ip, broker_port, 60)
File "/usr/local/lib/python3.9/dist-packages/paho/mqtt/client.py", line 914, in connect
return self.reconnect()
File "/usr/local/lib/python3.9/dist-packages/paho/mqtt/client.py", line 1044, in reconnect
sock = self._create_socket_connection()
File "/usr/local/lib/python3.9/dist-packages/paho/mqtt/client.py", line 3685, in _create_socket_connection
return socket.create_connection(addr, timeout=self._connect_timeout, source_address=source)
File "/usr/lib/python3.9/socket.py", line 843, in create_connection
raise err
File "/usr/lib/python3.9/socket.py", line 831, in create_connection
sock.connect(sa)
socket.timeout: timed out
I have tested to connect the second NIC in the NAS to a port in the switch tagged with IOT so the NAS gets an IP in IOT. Now the python client above connects the the MQTT server on the NAS.
I don’t understand. There is still something strange in my network.
This has nothing to do with DHCP, I am asking if you see the MAC address on pfsense arp table or mac address table.
You initially said that:
Your “main network” as you explained it has no vlan (or VLAN 1), not VLAN11 as you tried to go back from the rbpi to Synology who is also on network main (VLAN1). There is no VLAN11 in this. VLAN11 is admin network.
This has nothing to do with DHCP, I am asking if you see the MAC address on pfsense arp table or mac address table.
NAS ARP:
I have four rpis:
- housepi is in ARP table
- uthuspi is in ARP table
- uterumspi is in ARP table
- weatherstation is in ARP table
I think I have mixed things up with my naming. Here are my 5 networks in pfsense as described in my initial post:
- Main (no vlan)
- iot (vlan 88)
- guest (vlan 22)
- work (vlan 55)
- admin (vlan 11)
The network admin is actually called SKYNET. It looks like this in pfsense:
In Unifi Network Controller it loks like this:
Does your NAS have a gateway address defined and which is it?
From your main computer (Skynet network VLAN 11) can you reach your NAS?
When you have issues like this, start from pfsense and go the CLI and try to ping each devices in each network. If that works, as least your know the wiring and vlans are ok, but your rules (or maybe even NATing) is causing you trouble.
When you are on your rpi, it is using the IP of the device itself or does it have other virtual network on it? I ask because your firewall rules strictly enforce IP from the IOT network, so if you have another IP coming out of the rpi, it will be dropped.
The other thing to check (that ( said in my first reply): check the port on your unify switch where you rpi is connected: both PVID and untagged must be the same (in your case 11). If you have PVID 1 (defaut) and untagged 11, that will mean frames leaving the switch port going to you rpi will be on vlan 11, but frames coming from the rpi will be considered vlan 1.
Note: Tthat’s why I aim to not use network switch with PVID and Untagged vlan which is too error prone uselessly confusing. I think that comes from HP back in the day when they introduced their Procurve but I am not sure. Having just “native vlan” (or access vlan) and then allowed vlans (tagged vlans) is a much simpler way to understand and manage vlan-ing.
Does your NAS have a gateway address defined and which is it?
It is on network SKYNET.
From your main computer (Skynet network VLAN 11) can you reach your NAS?
Yes; via ssh, via web interface to DSM admin, …
When you have issues like this, start from pfsense and go the CLI and try to ping each devices in each network. If that works, as least your know the wiring and vlans are ok, but your rules (or maybe even NATing) is causing you trouble.
I ssh:ed into the pfsense and ping the NAS on SKYNET and RPI 1 / RPI 2 on IOT:
When you are on your rpi, it is using the IP of the device itself or does it have other virtual network on it? I ask because your firewall rules strictly enforce IP from the IOT network, so if you have another IP coming out of the rpi, it will be dropped.
I assume this shows what you ask for:
VNC is disabled.
The other thing to check (that ( said in my first reply): check the port on your unify switch where you rpi is connected: both PVID and untagged must be the same (in your case 11). If you have PVID 1 (defaut) and untagged 11, that will mean frames leaving the switch port going to you rpi will be on vlan 11, but frames coming from the rpi will be considered vlan 1.
I’m not sure how to check this. PVID is a term I have not configured. After googling I assume the following info provide my setup.
Port config in Unifi Network Controller
VLAN config:
Note: Tthat’s why I aim to not use network switch with PVID and Untagged vlan which is too error prone uselessly confusing. I think that comes from HP back in the day when they introduced their Procurve but I am not sure. Having just “native vlan” (or access vlan) and then allowed vlans (tagged vlans) is a much simpler way to understand and manage vlan-ing.
Is it my way of configuring VLANs that complicates things or is it the way Unifi handles VLANs?