This post is meant to share a bizarre issue that we haven’t encountered before, for the amusement of anyone who is like me and enjoys such things. If anyone has knowledgeable suggestions I’m all ears, but we’ve already written this issue off as not worth spending additional time on.
The firewall in question is an existing install with 579 days of uptime. At the beginning of this tale it was on 2.3.2 and during troubleshooting it was upgraded to 2.4.4_1. The hardware is a Supermicro C2758 with 4 NICs, I don’t know more than that right now. PFSense is installed on the bare metal.
On to the issue: We had reason to add new VLANs to the firewall’s LAN port. Created VLANs, assigned interfaces and static IPs, created basic “allow all from XXX net” firewalls rules on each one. And then we tried to reach things in the subnets… fail. VLANs tagged properly on switches? Check. Devices actually on the IPs we think they’re on? Check, things within these subnets/VLANs can ping each other, but not the firewall, and the firewall can’t ping them. What about ARP? That is working. DHCP? Clients in the VLANs can get IPs from the DHCP server on PFSense. ARP and DHCP but nothing else usually means firewall rule issue? Remade firewall rules (and made sure everything was applied), no dice. Changed firewall rules to “Allow all from any”, no dice. Disabled firewall completely with “pfctl -d”, still can’t get anything through.
The existing VLANs did and do work fine, but all of the new ones we create have the issue above. Our only next step is to backup, wipe, and reinstall, but sadly there isn’t the budget nor time allotted for doing that, as this isn’t a local system for us. Our decision is that this one is just stuck with the VLANs it has until the customer wants to upgrade their whole network which would include a new firewall. They’ll have to eventually, because this is a Starwood hotel and PFSense (even NetGate appliances) isn’t Marriott approved.
Hopefully this issue gave some amusement and thought puzzling.