Vlan setup with PBR to multiple WANs on EdgeRouter

Hi. I have been trying to setup my network for the past 2 weeks now and I exhausted all of my options multiple times at this point. If there is any other info you might need that I didn’t post please ask…

My Setup

This is for a small business network. I want to use lan 1 for primary office, vlan 10 for guest, vlan 20 for voip and vlan 30 for a independant connection for Zac(Lives on premise in a rv trailer atm and wants his own internet connection).
I followed https://help.ui.com/hc/en-us/articles/204952274-EdgeRouter-Policy-based-routing-source-address-based- to setup my initial policies. I made a modification to use a interface to route to instead of an ip address.

set protocols static table 1 interface-route 0.0.0.0/0 next-hop-interface eth0
set protocols static table 10 interface-route 0.0.0.0/0 next-hop-interface eth0
set protocols static table 20 interface-route 0.0.0.0/0 next-hop-interface eth1
set protocols static table 30 interface-route 0.0.0.0/0 next-hop-interface eth2

Goal

Primary office connection and guest network should route to eth0.
Voip route to eth1
Zacs vlan 30 to eth2
Eth7 is used as the trunk port to my unifi network.

The unifi network has these networks setup. guest-10, LAN, voip-20 and zac-30.

The following image is the basic network structure I am after.

Equipment

EdgeRouter

Version: v2.0.8-hotfix.1
HW model: EdgeRouter 8-Port
Uptime: 16:47:37 up 32 min, 1 user, load average: 0.01, 0.03, 0.06

Primary Unfi Switch

Version: 4.3.21.11325
Model: USW-24-POE

Other Unifi Gear

  • USW-Flex-Mini switch
  • US-8-60W switch
  • UAP-Pro LTS
  • UAP-Outdoor+ LTS
  • UAP-AC-Pro
  • UAP-AC-M-Pro
  • UAP-AC-M

My Problem

So I have the edgerouter and the unifi network setup and it seems to be working alright as long as I only have the primary internet connected. As soon as I connect the modem for Zacs internet, the vlan 30 loses internet. If that internet is the only one connected the main network and guest network fallback to Zacs internet as expected. At the same time Zacs internet falls back on the primary internet if that modem isn’t present.

Also the guest network (vlan 10) only seems to keep internet as long as I don’t enable the PBR that I have setup. set interfaces ethernet eth7 vif 10 firewall in modify PBR running that command disconnects that internet connection from vlan 10… Yet running that same command on Zacs network (vlan 30) has no different effects and keeps working the same way.

Finally here is my boot config file for the edgerouter.

    firewall {
    all-ping enable
    broadcast-ping disable
    group {
        network-group vlans {
            network 192.168.10.0/24
            network 192.168.20.0/24
            network 192.168.30.0/24
        }
    }
    ipv6-name WANv6_IN {
        default-action drop
        description "WAN inbound traffic forwarded to LAN"
        enable-default-log
        rule 10 {
            action accept
            description "Allow established/related sessions"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
    }
    ipv6-name WANv6_LOCAL {
        default-action drop
        description "WAN inbound traffic to the router"
        enable-default-log
        rule 10 {
            action accept
            description "Allow established/related sessions"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
        rule 30 {
            action accept
            description "Allow IPv6 icmp"
            protocol ipv6-icmp
        }
        rule 40 {
            action accept
            description "allow dhcpv6"
            destination {
                port 546
            }
            protocol udp
            source {
                port 547
            }
        }
    }
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians enable
    modify PBR {
        rule 1 {
            action modify
            description lan1
            modify {
                table 1
            }
            source {
                address 192.168.1.0/24
            }
        }
        rule 2 {
            action modify
            description inter-vlan
            destination {
                group {
                    network-group vlans
                }
            }
            modify {
                table main
            }
        }
        rule 10 {
            action modify
            description vlan10
            modify {
                table 10
            }
            source {
                address 192.168.10.0/24
            }
        }
        rule 20 {
            action modify
            description vlan20
            modify {
                table 20
            }
            source {
                address 192.168.20.0/24
            }
        }
        rule 30 {
            action modify
            description vlan30
            modify {
                table 30
            }
            source {
                address 192.168.30.0/24
            }
        }
    }
    name WAN_IN {
        default-action drop
        description "WAN to internal"
        rule 10 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
    }
    name WAN_LOCAL {
        default-action drop
        description "WAN to router"
        rule 10 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
    }
    receive-redirects disable
    send-redirects enable
    source-validation disable
    syn-cookies enable
}
interfaces {
    ethernet eth0 {
        address dhcp
        description Internet
        duplex auto
        firewall {
            in {
                ipv6-name WANv6_IN
                name WAN_IN
            }
            local {
                ipv6-name WANv6_LOCAL
                name WAN_LOCAL
            }
        }
        speed auto
    }
    ethernet eth1 {
        address dhcp
        description Voip
        duplex auto
        speed auto
    }
    ethernet eth2 {
        address dhcp
        description Zac
        disable
        duplex auto
        speed auto
    }
    ethernet eth3 {
        duplex auto
        speed auto
    }
    ethernet eth4 {
        duplex auto
        speed auto
    }
    ethernet eth5 {
        duplex auto
        speed auto
    }
    ethernet eth6 {
        address 192.168.5.1/24
        description Admin
        duplex auto
        speed auto
    }
    ethernet eth7 {
        address 192.168.1.1/24
        description Primary
        duplex auto
        speed auto
        vif 1 {
            disable
            firewall {
                in {
                    modify PBR
                }
            }
        }
        vif 10 {
            address 192.168.10.1/24
            description "Guest - 10"
            firewall {
                in {
                }
            }
            mtu 1500
        }
        vif 20 {
            address 192.168.20.1/24
            description "voip - 20"
            firewall {
                in {
                    modify PBR
                }
            }
            mtu 1500
        }
        vif 30 {
            address 192.168.30.1/24
            description "zac - 30"
            firewall {
                in {
                    modify PBR
                }
            }
            mtu 1500
        }
    }
    loopback lo {
        ip {
            ospf {
                authentication {
                }
                dead-interval 40
                hello-interval 10
                priority 1
                retransmit-interval 5
                transmit-delay 1
            }
        }
    }
}
protocols {
    static {
        route 0.0.0.0/0 {
            next-hop 24.142.39.171 {
            }
            next-hop 24.215.83.184 {
            }
        }
        table 1 {
            interface-route 0.0.0.0/0 {
                next-hop-interface eth0 {
                }
            }
        }
        table 10 {
            interface-route 0.0.0.0/0 {
                next-hop-interface eth0 {
                }
            }
        }
        table 20 {
            interface-route 0.0.0.0/0 {
                next-hop-interface eth1 {
                }
            }
        }
        table 30 {
            interface-route 0.0.0.0/0 {
                next-hop-interface eth2 {
                }
            }
        }
    }
}
service {
    dhcp-server {
        disabled false
        hostfile-update disable
        shared-network-name LAN {
            authoritative disable
            subnet 192.168.1.0/24 {
                default-router 192.168.1.1
                dns-server 192.168.1.1
                lease 86400
                start 192.168.1.6 {
                    stop 192.168.1.254
                }
                static-mapping TREVOR-PC {
                    ip-address 192.168.1.5
                    mac-address f4:8e:38:af:6a:cb
                }
                static-mapping nas1 {
                    ip-address 192.168.1.6
                    mac-address 80:61:5f:08:20:7f
                }
                static-mapping unifi-server {
                    ip-address 192.168.1.93
                    mac-address b8:27:eb:e2:85:ce
                }
            }
        }
        shared-network-name LAN_ADMIN {
            authoritative disable
            subnet 192.168.5.0/24 {
                default-router 192.168.5.1
                dns-server 192.168.5.1
                lease 86400
                start 192.168.5.15 {
                    stop 192.168.5.254
                }
            }
        }
        shared-network-name guest-vlan {
            authoritative disable
            subnet 192.168.10.0/24 {
                default-router 192.168.10.1
                dns-server 192.168.10.1
                lease 86400
                start 192.168.10.6 {
                    stop 192.168.10.254
                }
            }
        }
        shared-network-name voip-vlan {
            authoritative disable
            subnet 192.168.20.0/24 {
                default-router 192.168.20.1
                dns-server 192.168.20.1
                lease 86400
                start 192.168.20.6 {
                    stop 192.168.20.254
                }
            }
        }
        shared-network-name zac-vlan {
            authoritative disable
            subnet 192.168.30.0/24 {
                default-router 192.168.30.1
                dns-server 192.168.30.1
                lease 86400
                start 192.168.30.6 {
                    stop 192.168.30.254
                }
            }
        }
        static-arp disable
        use-dnsmasq disable
    }
    dns {
        forwarding {
            cache-size 10000
            force-public-dns-boost
            listen-on eth6
            listen-on eth7
            listen-on eth7.10
            listen-on eth7.20
            listen-on eth7.30
        }
    }
    gui {
        http-port 80
        https-port 443
        older-ciphers enable
    }
    nat {
        rule 5010 {
            description "masquerade for WAN"
            outbound-interface eth0
            type masquerade
        }
        rule 5011 {
            description "masquerade for zacs WAN"
            log disable
            outbound-interface eth2
            protocol all
            type masquerade
        }
    }
    ssh {
        port 22
        protocol-version v2
    }
}
system {
    host-name EdgeRouter-8-Port
    login {
        user trevor {
            authentication {
                encrypted-password I-Changed-This
            }
            level admin
        }
    }
    ntp {
        server 0.ubnt.pool.ntp.org {
        }
        server 1.ubnt.pool.ntp.org {
        }
        server 2.ubnt.pool.ntp.org {
        }
        server 3.ubnt.pool.ntp.org {
        }
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone UTC
}


/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "config-management@1:conntrack@1:cron@1:dhcp-relay@1:dhcp-server@4:firewall@5:ipsec@5:nat@3:qos@1:quagga@2:suspend@1:system@4:ubnt-pptp@1:ubnt-udapi-server@1:ubnt-unms@1:ubnt-util@1:vrrp@1:vyatta-netflow@1:webgui@1:webproxy@1:zone-policy@1" === */
/* Release version: v2.0.8-hotfix.1.5278088.200305.1641 */

Any thoughts, questions and concerns are appreciated.

Thanks for the read,
Have a great day.

A very confused Trevor

So I just noticed that the firewall wasn’t setup the same for eth2 (zacs internet - vlan 30) as it was for our primary one. The primary one I used the wizard to create and the eth1 and eth2 I setup manually. So I updated the firewall to match that of eth0.

set interfaces ethernet eth2 firewall in ipv6-name WANv6_IN
set interfaces ethernet eth2 firewall in name WAN_IN
set interfaces ethernet eth2 firewall local ipv6-name WANv6_LOCAL
set interfaces ethernet eth2 firewall local name WAN_LOCAL

Unfortunately nothing changed with that.
Another change that I set was I set both of eth0 and eth2 to a static ip. Same results still.