Weird vlan and network issue, need suggestions on where to look

Networking & Firewalls
1

hello everyone,

I am having a weird vlan / network issue that is really messing with my head. any suggestions on where to look are greatly appreciated.

network setup
cable modem (passthrough) to pfsense to ubiquiti US-48-750W to US-48

home lab servers are on US-48 (NO POE)
switches in other rooms and AP’s are on US-48-750W
all vlans work fine with one weird exception

vlan 10 - home x.x.10.0/24
vlan 20 - guest x.x.20.0/24
vlan 30 - entartainment x.x.30.0/24
vlan 40 - IOT x.x.40.0/24
vlan 50 - servers x.x.50.0/24
vlan 60 - lab x.x.60.0/24
vlan 70 - management x.x.70.0/24
vlan 80 - video x.x.80.0/24
vlan 90 - voip (future) x.x.90.0/24

currently IOT/ENT/GUEST can only talk to WAN, not able to talk to anything else inside
have a couple of things inside servers, synology, vmware servers, etc
from home, lab, and management networks i can talk to servers just fine
from servers i can talk to home, lab and management just fine

I CANNOT TALK FROM SERVER to SERVER. even the PFSense diagnostics ping can not ping anything in the server network from the server network. but anything on another network can ping anything on the server network.

very confused, all other networks run just fine and I haven’t fully setup firewall rules yet.
any suggestions on where to look would be great.

Thank you.

2

When properly setup the firewall controls access between the VLANs so check your rules to make sure they allow that.

3

Tom,

Thank you for the reply, I did check that before, and between vlans is fine, just within the same vlan is a problem, and only on the 1 vlan.

Thank you.

4

Traffic within a VLAN (within any subnet, aka a Layer 2 broadcast domain) is not handled by the firewall. Devices will always try to reach other IPs in their same subnet directly, only traffic for which a route has been created (which includes the default gateway as the route of last resort) are sent to an intermediate IP address. So for your servers that can’t communicate with each other, you need to make sure their IPs and subnet masks are really what you think they are. After trying to ping one server from another, check the ARP tables on the servers to see if they learned each other’s MAC address (run “arp -a” on almost any OS to see this)

5

Make sure your switch is configured properly too.

6

brwainer,

Thank you for the reply, using your info I did verify the IP’s and subnet masks were correct. I had a feeling that they would be due to all devices are set for DHCP with static mappings within the DHCP Server. this still was a problem until- due to unforseen circumstances of a neiborhood wide power outage- i had to shut everything down because my UPS was getting low on battery. Once I brought everything back up this morning, everything works just fine.

This is still confusing to me and a part of me hopes it happens again so I can dig in and troubleshoot more. the more we troubleshoot the more we learn.

to put into more detail what was happening, I have 9 vlans that are broken down as follows:
vlan 10 - home (PC’s, Tablets, etc…)
vlan 20 - guest (kids friends that come over)
vlan 30 - entertainment (game systems, tv’s etc)
vlan 40 - IOT (google homes, the monitoring device for my smoker, smart home devices, etc…)
vlan 50 - servers/NAS (synology nas, esxi hosts, etc…)
vlan 60 - LAB (my play space)
vlan 70 - MGT (switches, pfsense management, server consoles(idrac, ilom)
vlan 80 - video (security cameras)
vlan 90 - voip (future plans for playing with voip in the house)

firewall rules are set to the following:
vlan 10 - HOME - access to everything except management network
vlan 20 - guest - access to WAN only
vlan 30 - entertainment - Access to WAN Only- UPNP enabled
vlan 40 - IOT - Access to WAN only- mdns enabled
vlan 50 - server/nas - access to everything except management network
vlan 60 - LAB - access to wan only
vlan 70 - management - no access outside of management network
vlan 80 - video - no access outside of video network
vlan 90 - voip - currently no access outside of voip network- until i get phones to play with.

from the home network (vlan 10) i was able to have full access to everything in the server network I could bring up esxi and synology front ends, ping them without a problem.
the ping function from the pfsense was also able to ping all server/nas devices when you used the home network (vlan 10)
the ping function from the pfsense was NOT able to ping all server/nas devices from the server network (vlan 50)
the synology could NOT ping the pfsense, or any of the esxi servers, the esxi servers could NOT ping the pfsense or the synology. all ip’s in the same network, with the proper /24 range (DHCP assigned with static mapping) they would get their dhcp lease without a problem.) but from the home network I was able to access all of these devices in vlan 50.

sorry for the long post, yes everything works now that the entire system has been rebooted so the problem is solved, I just wish i knew what the problem was in the first place.

Thank you everyone for all your advice.

7

Are you sure you have the server VLAN tagged on your trunk port between US-48-750W to US-48?