I seem to be missing something getting this to work correctly. I know this setup is wild and I will get some flack for attempting it; but its been my mission to get it to work properly. I think it’s possible that STP is causing issues but unsure.
I have two switches, a US-16-XG and a US-16-150W. They are connected via 4 aggregated trunk ports.
On the US-16-XG I have my two proxmox hosts connected via 10G DAC. On the US-16-150W, I have WAN connected on aggregated ports 1 and 2 on vlan 100.
Now for my proxmox hosts. They both have a vlan-aware bridge on the 10Gb port.
Node 1 has pfsense running virtualized with two nics. Both nics are connected to that vmbr0 bridge but WAN is on vlan 100 and LAN is trunk.
Also on Node1 is two domain controllers that run DHCP and DNS.
Now for pfsense. I have vlans:
10 - 10.0.10.0/24 with gateway 10.0.10.254
20 - 10.0.20.0/24 with gateway 10.0.20.254
30 - 10.0.30.0/24 with gateway 10.0.30.254
40 - 10.0.40.0/24 with gateway 10.0.40.254
50 - 10.0.50.0/24 with gateway 10.0.50.254
200 - 10.0.200.0/24 with gateway 10.0.200.254
Each has a corresponding scope setup on the domain controllers DHCP.
Currently i’m only using vlan 10, 30 and 40. 10 is servers, 30 is users, and 40 is IoT.
10 and 30 has an all/all rule setup in the firewall. 40 blocks all except internet and DNS.
I also have all these vlans added to Unifi controller.
Current issues:
Second proxmox node vms can’t reach network
DNS sometimes cuts out but I think it’s because my second DC was living on the second node and I have DNS as a 50/50 split. Has been working fine since moving it to node1.
Should these vlans be added to unifi as networks or vlans only? I use 10.0.0.0/24 as my base network. Proxmox nodes, switches, access point all live on this network.
It mostly works, I have my wifi network on vlan 30. My laptop can reach DHCP and DNS. I’m writing this now on my laptop. I guess my main issue is when I create a VM on my second proxmox node with a nic on any vlan. It can’t reach DNS, its own gateway, or outside internet. It can however hit up DHCP and request a lease.
Here is the packets captured from my testvm on node2 when trying to ping the gateway. You can see DHCP handed it 10.0.10.131 on the server vlan.
My gameplan is to have pfsense in HA mode where it can live migrate between node 1 and 2 in the event I need to take one down. The mac address on the wan nic won’t change so it shouldn’t have issues with my modem. Just need to figure out why node2 is having network issues.
