Complex network - DHCP requests for a specific VLAN not reaching server

Hi,

I have a complex network, with an HP core switch, several HP switches and UniFi switches and APs.

HP switches are configured manually via ssh.

UniFi infrastructure is handled by UniFi OS 5.0.6 with UniFi Network 10.3.55 running in a Debian VM.

There is a pfSense firewall that handles DHCP for the VLANs that need it.

I have quite a bunch of VLANs working this way.

I just added another one almost copying and pasting the configuration of one that is working, however DHCPDISCOVER (broadcast) packages from this specific VLAN don’t seem to be reaching the pfSense server (I don’t see them in the log).

My last check consisted on wiring an access port on this VLAN to my notebook (so as to remove any wifi related problem).

First I manually set up an IP address (192.168.248.99) in the network assigned to that VLAN (192.168.248.0/24).

This works OK and I can ping the core switch (192.168.248.1) and also de pfSense (192.168.248.13). So this seems to show that the VLAN is reaching from the notebook to the server.

However, when I try to run dhclient to get an address, it times out.

Any help would be really appreciated.

Did you set up firewall rule(s) in PFSense for the incoming traffic from the new VLAN? PFSense starts all new interfaces (networks) with no rules meaning it drops everything.

1 Like

You also need to make sure you have the newly created VLAN enabled and passing on the switch(es).

1 Like

did you configure pfsense to offer DHCP services on that VLAN?

1 Like

Yes, I know. However, I added this VLAN interface (LAN_CAM_IF) to an interface group that already had those rules enabled:

Yes. This was what I most time put double and triple checking, and the test I did seems to show that the VLAN traverses all those switches (192.168.248.13 is the firewall):

Yes:

What do the pfsense DHCP logs say? Any DORA request on that subnet?

Might need to run a packet capture. See if client is giving out a request.

There has been reported bugs with Kea DHCP in some versions. Not sure what version of pfsense you are running or if you have system patches updated.

1 Like

I don’t know what a DORA request is or how to look for it.

There is nothing in the logs for the range of this pool:

# grep 192.168.248 /var/log/dhcpd.log*
#

There are lots of entries for every pool, including this one that I don’t know what they mean, but it’s probably just a routine check:

# grep opt18 /var/log/dhcpd.log*
...
/var/log/dhcpd.log.6:Apr 30 15:14:35 fw-ecuador-00 kea-dhcp4[26069]: INFO  [kea-dhcp4.dhcpsrv.0x277c2cbdcd00] EVAL_RESULT Expression pool_opt18_0 evaluated to 1
/var/log/dhcpd.log.6:Apr 30 15:14:38 fw-ecuador-00 kea-dhcp4[26069]: INFO  [kea-dhcp4.dhcpsrv.0x277c2cbdcd00] EVAL_RESULT Expression pool_opt18_0 evaluated to 1
/var/log/dhcpd.log.6:Apr 30 15:14:44 fw-ecuador-00 kea-dhcp4[26069]: INFO  [kea-dhcp4.dhcpsrv.0x277c2cbdcd00] EVAL_RESULT Expression pool_opt18_0 evaluated to 1
/var/log/dhcpd.log.6:Apr 30 15:14:46 fw-ecuador-00 kea-dhcp4[26069]: INFO  [kea-dhcp4.dhcpsrv.0x277c2cbdcd00] EVAL_RESULT Expression pool_opt18_0 evaluated to 1
/var/log/dhcpd.log.6:Apr 30 15:14:47 fw-ecuador-00 kea-dhcp4[26069]: INFO  [kea-dhcp4.dhcpsrv.0x277c2cbdcd00] EVAL_RESULT Expression pool_opt18_0 evaluated to 1
#

If I check for the MAC address of my notebook wired interface which I used for testing, I only see a couple of tests I did changing the VLAN port to check that it worked, but nothing when using the VLAN I just created:

# grep ac:1a:3d:11:4f:1d dhcpd.log*
dhcpd.log.4:Apr 30 16:17:16 fw-ecuador-00 kea-dhcp4[26069]: INFO  [kea-dhcp4.leases.0x277c2cbde200] DHCP4_LEASE_ADVERT [hwtype=1 ac:1a:3d:11:4f:1d], cid=[01:ac:1a:3d:11:4f:1d], tid=0xfcd29e70: lease 192.168.136.2 will be advertised
dhcpd.log.4:Apr 30 16:17:43 fw-ecuador-00 kea-dhcp4[26069]: INFO  [kea-dhcp4.leases.0x277c2cbde200] DHCP4_LEASE_ADVERT [hwtype=1 ac:1a:3d:11:4f:1d], cid=[01:ac:1a:3d:11:4f:1d], tid=0xa00ec5e: lease 192.168.136.3 will be advertised
dhcpd.log.4:Apr 30 16:17:43 fw-ecuador-00 kea-dhcp4[26069]: INFO  [kea-dhcp4.leases.0x277c2cbde200] DHCP4_LEASE_ALLOC [hwtype=1 ac:1a:3d:11:4f:1d], cid=[01:ac:1a:3d:11:4f:1d], tid=0xa00ec5e: lease 192.168.136.3 has been allocated for 7200 seconds
dhcpd.log.4:Apr 30 16:48:38 fw-ecuador-00 kea-dhcp4[26069]: INFO  [kea-dhcp4.leases.0x277c2cbe1a00] DHCP4_LEASE_ADVERT [hwtype=1 ac:1a:3d:11:4f:1d], cid=[no info], tid=0xe5652f3e: lease 192.168.136.3 will be advertised
dhcpd.log.4:Apr 30 16:48:41 fw-ecuador-00 kea-dhcp4[26069]: INFO  [kea-dhcp4.leases.0x277c2cbe1a00] DHCP4_LEASE_ADVERT [hwtype=1 ac:1a:3d:11:4f:1d], cid=[no info], tid=0xe5652f3e: lease 192.168.136.3 will be advertised
dhcpd.log.4:Apr 30 16:48:46 fw-ecuador-00 kea-dhcp4[26069]: INFO  [kea-dhcp4.leases.0x277c2cbe1a00] DHCP4_LEASE_ADVERT [hwtype=1 ac:1a:3d:11:4f:1d], cid=[no info], tid=0xe5652f3e: lease 192.168.136.3 will be advertised
dhcpd.log.4:Apr 30 16:48:46 fw-ecuador-00 kea-dhcp4[26069]: INFO  [kea-dhcp4.leases.0x277c2cbe1a00] DHCP4_LEASE_ALLOC [hwtype=1 ac:1a:3d:11:4f:1d], cid=[no info], tid=0xe5652f3e: lease 192.168.136.3 has been allocated for 7200 seconds
dhcpd.log.4:Apr 30 16:49:07 fw-ecuador-00 kea-dhcp4[26069]: INFO  [kea-dhcp4.leases.0x277c2cbe1300] DHCP4_RELEASE [hwtype=1 ac:1a:3d:11:4f:1d], cid=[no info], tid=0x246d2234: address 192.168.136.3 was released properly.
dhcpd.log.4:Apr 30 16:49:07 fw-ecuador-00 kea-dhcp4[26069]: INFO  [kea-dhcp4.leases.0x277c2cbe1300] DHCP4_RELEASE_EXPIRED [hwtype=1 ac:1a:3d:11:4f:1d], cid=[no info], tid=0x246d2234: address 192.168.136.3 expired on release.

“DORA”: Discover, Offer, Request, Ack.

HP core switch. Is this layer 3? could be a broadcast forwarding issue. Do you have DHCP relay setup/IP helper setup on the switch?

1 Like

could you bounce kea dhcp? (restart)

Post Diagnostics | Routes | IPV4 routes

I saw earlier that you route default to 248.1 but pfsense is 248.13

1 Like

It is layer 3, but I don’t need to enable forwarding since the pfSense is connected via TRUNK port to every (actually most) VLAN.

DHCP is happily working on at least half a dozen other VLANs with the same setup.

I can try restarting KEA (I think I did that when I set it up, but now I’m not sure). Will try tomorrow that I’ll be at the office and will be able to test it.

Regarding the routes: 248.1 points to the L3 switch which handles internal routing (with ACLs) and has a default route to one of the pfSense interfaces. 248.13 is the pfSense.

output of ip route show

I understand you want to see the routing table from the pfSense firewall… there’s no ip command in FreeBSD, this is the output of netstat -rn4 (with public IPs redacted):

Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            RDACTED.128.94     UGS        bge0
10.1.2.0/23        link#31            U        bge2.2
10.1.2.13          link#6             UHS         lo0
10.11.12.0/22      link#14            U        bge2.3
10.11.12.13        link#6             UHS         lo0
10.13.13.0/24      link#18            U      bge2.351
10.13.13.13        link#6             UHS         lo0
10.20.30.0/24      link#22            U      bge2.113
10.20.30.13        link#6             UHS         lo0
10.31.128.0/20     link#37            U       bge2.18
10.31.128.13       link#6             UHS         lo0
10.32.128.0/20     link#34            U       bge2.19
10.32.128.13       link#6             UHS         lo0
10.40.0.0/24       link#13            U      bge2.400
10.40.0.13         link#6             UHS         lo0
10.63.128.0/20     link#36            U       bge2.16
10.63.128.13       link#6             UHS         lo0
10.64.128.0/20     link#33            U       bge2.17
10.64.128.13       link#6             UHS         lo0
10.100.200.0/24    link#27            U        ovpns2
10.100.200.1       link#6             UHS         lo0
10.100.201.0/24    link#29            U        ovpns5
10.100.201.1       link#6             UHS         lo0
10.101.208.0/22    link#28            U        ovpns4
10.101.208.1       link#6             UHS         lo0
10.101.216.0/22    link#26            U        ovpns1
10.101.216.1       link#6             UHS         lo0
10.127.128.0/20    link#35            U       bge2.12
10.127.128.13      link#6             UHS         lo0
10.128.128.0/20    link#32            U       bge2.13
10.128.128.13      link#6             UHS         lo0
127.0.0.1          link#6             UH          lo0
REDACTD.0.0/16     REDACTD.78.1       UGS        bge1
REDACTD.78.0/24    link#2             U          bge1
REDACTD.78.3       link#6             UHS         lo0
172.19.128.0/20    link#23            U      bge2.112
172.19.128.13      link#6             UHS         lo0
172.19.192.0/20    link#19            U      bge2.212
172.19.192.13      link#6             UHS         lo0
172.20.128.0/20    link#24            U      bge2.111
172.20.128.13      link#6             UHS         lo0
172.20.192.0/20    link#20            U      bge2.211
172.20.192.13      link#6             UHS         lo0
172.27.100.0/24    link#11            U      bge2.100
172.27.100.13      link#6             UHS         lo0
172.27.101.0/24    link#12            U      bge2.101
172.27.101.13      link#6             UHS         lo0
172.27.128.0/18    link#17            U      bge2.128
172.27.128.13      link#6             UHS         lo0
192.168.128.0/20   link#25            U      bge2.110
192.168.128.13     link#6             UHS         lo0
192.168.192.0/20   link#21            U      bge2.210
192.168.192.13     link#6             UHS         lo0
192.168.208.0/22   link#30            U        ovpns6
192.168.208.1      link#6             UHS         lo0
192.168.248.0/24   link#38            U      bge2.250
192.168.248.13     link#6             UHS         lo0
REDCTD.122.128/28  link#1             U          bge0
REDCTD.122.141     link#6             UHS         lo0
RDACTED.128.64/27  link#1             U          bge0
RDACTED.128.93     link#6             UHS         lo0

The problematic VLAN is bge2.250. VLANs bge2.110 and bge2.210 are configured like bge2.250 and work OK.

FWIW, this is the routing table in the HP layer 3 switch:

<sw-core>display ip routing-table 

Destinations : 97	Routes : 97

Destination/Mask    Proto  Pre  Cost         NextHop         Interface
0.0.0.0/0           Static 60   0            172.27.254.2    Vlan254
0.0.0.0/32          Direct 0    0            127.0.0.1       InLoop0
10.1.2.0/23         Direct 0    0            10.1.2.1        Vlan2
10.1.2.0/32         Direct 0    0            10.1.2.1        Vlan2
10.1.2.1/32         Direct 0    0            127.0.0.1       InLoop0
10.1.3.255/32       Direct 0    0            10.1.2.1        Vlan2
10.11.12.0/22       Direct 0    0            10.11.12.1      Vlan3
10.11.12.0/32       Direct 0    0            10.11.12.1      Vlan3
10.11.12.1/32       Direct 0    0            127.0.0.1       InLoop0
10.11.15.255/32     Direct 0    0            10.11.12.1      Vlan3
10.13.13.0/24       Direct 0    0            10.13.13.1      Vlan3517
10.13.13.0/32       Direct 0    0            10.13.13.1      Vlan3517
10.13.13.1/32       Direct 0    0            127.0.0.1       InLoop0
10.13.13.255/32     Direct 0    0            10.13.13.1      Vlan3517
10.20.30.0/24       Direct 0    0            10.20.30.1      Vlan113
10.20.30.0/32       Direct 0    0            10.20.30.1      Vlan113
10.20.30.1/32       Direct 0    0            127.0.0.1       InLoop0
10.20.30.255/32     Direct 0    0            10.20.30.1      Vlan113
10.31.128.0/20      Direct 0    0            10.31.128.1     Vlan18
10.31.128.0/32      Direct 0    0            10.31.128.1     Vlan18
10.31.128.1/32      Direct 0    0            127.0.0.1       InLoop0
10.31.143.255/32    Direct 0    0            10.31.128.1     Vlan18
10.32.128.0/20      Direct 0    0            10.32.128.1     Vlan19
10.32.128.0/32      Direct 0    0            10.32.128.1     Vlan19
10.32.128.1/32      Direct 0    0            127.0.0.1       InLoop0
10.32.143.255/32    Direct 0    0            10.32.128.1     Vlan19
10.63.128.0/20      Direct 0    0            10.63.128.1     Vlan16
10.63.128.0/32      Direct 0    0            10.63.128.1     Vlan16
10.63.128.1/32      Direct 0    0            127.0.0.1       InLoop0
10.63.143.255/32    Direct 0    0            10.63.128.1     Vlan16
10.64.128.0/20      Direct 0    0            10.64.128.1     Vlan17
10.64.128.0/32      Direct 0    0            10.64.128.1     Vlan17
10.64.128.1/32      Direct 0    0            127.0.0.1       InLoop0
10.64.143.255/32    Direct 0    0            10.64.128.1     Vlan17
10.100.200.0/23     Static 60   0            10.13.13.13     Vlan3517
10.101.208.0/22     Static 60   0            10.13.13.13     Vlan3517
10.101.216.0/22     Static 60   0            10.13.13.13     Vlan3517
10.127.128.0/20     Direct 0    0            10.127.128.1    Vlan12
10.127.128.0/32     Direct 0    0            10.127.128.1    Vlan12
10.127.128.1/32     Direct 0    0            127.0.0.1       InLoop0
10.127.143.255/32   Direct 0    0            10.127.128.1    Vlan12
10.128.128.0/20     Direct 0    0            10.128.128.1    Vlan13
10.128.128.0/32     Direct 0    0            10.128.128.1    Vlan13
10.128.128.1/32     Direct 0    0            127.0.0.1       InLoop0
10.128.143.255/32   Direct 0    0            10.128.128.1    Vlan13
127.0.0.0/8         Direct 0    0            127.0.0.1       InLoop0
127.0.0.0/32        Direct 0    0            127.0.0.1       InLoop0
127.0.0.1/32        Direct 0    0            127.0.0.1       InLoop0
127.255.255.255/32  Direct 0    0            127.0.0.1       InLoop0
172.19.128.0/20     Direct 0    0            172.19.128.1    Vlan112
172.19.128.0/32     Direct 0    0            172.19.128.1    Vlan112
172.19.128.1/32     Direct 0    0            127.0.0.1       InLoop0
172.19.143.255/32   Direct 0    0            172.19.128.1    Vlan112
172.19.192.0/20     Direct 0    0            172.19.192.1    Vlan212
172.19.192.0/32     Direct 0    0            172.19.192.1    Vlan212
172.19.192.1/32     Direct 0    0            127.0.0.1       InLoop0
172.19.207.255/32   Direct 0    0            172.19.192.1    Vlan212
172.20.128.0/20     Direct 0    0            172.20.128.1    Vlan111
172.20.128.0/32     Direct 0    0            172.20.128.1    Vlan111
172.20.128.1/32     Direct 0    0            127.0.0.1       InLoop0
172.20.143.255/32   Direct 0    0            172.20.128.1    Vlan111
172.20.192.0/20     Direct 0    0            172.20.192.1    Vlan211
172.20.192.0/32     Direct 0    0            172.20.192.1    Vlan211
172.20.192.1/32     Direct 0    0            127.0.0.1       InLoop0
172.20.207.255/32   Direct 0    0            172.20.192.1    Vlan211
172.27.100.0/24     Direct 0    0            172.27.100.1    Vlan100
172.27.100.0/32     Direct 0    0            172.27.100.1    Vlan100
172.27.100.1/32     Direct 0    0            127.0.0.1       InLoop0
172.27.100.255/32   Direct 0    0            172.27.100.1    Vlan100
172.27.101.0/24     Direct 0    0            172.27.101.1    Vlan101
172.27.101.0/32     Direct 0    0            172.27.101.1    Vlan101
172.27.101.1/32     Direct 0    0            127.0.0.1       InLoop0
172.27.101.255/32   Direct 0    0            172.27.101.1    Vlan101
172.27.128.0/18     Direct 0    0            172.27.128.1    Vlan128
172.27.128.0/32     Direct 0    0            172.27.128.1    Vlan128
172.27.128.1/32     Direct 0    0            127.0.0.1       InLoop0
172.27.191.255/32   Direct 0    0            172.27.128.1    Vlan128
172.27.254.0/24     Direct 0    0            172.27.254.1    Vlan254
172.27.254.0/32     Direct 0    0            172.27.254.1    Vlan254
172.27.254.1/32     Direct 0    0            127.0.0.1       InLoop0
172.27.254.255/32   Direct 0    0            172.27.254.1    Vlan254
192.168.128.0/20    Direct 0    0            192.168.128.1   Vlan110
192.168.128.0/32    Direct 0    0            192.168.128.1   Vlan110
192.168.128.1/32    Direct 0    0            127.0.0.1       InLoop0
192.168.143.255/32  Direct 0    0            192.168.128.1   Vlan110
192.168.192.0/20    Direct 0    0            192.168.192.1   Vlan210
192.168.192.0/32    Direct 0    0            192.168.192.1   Vlan210
192.168.192.1/32    Direct 0    0            127.0.0.1       InLoop0
192.168.207.255/32  Direct 0    0            192.168.192.1   Vlan210
192.168.208.0/22    Static 60   0            10.13.13.13     Vlan3517
192.168.248.0/24    Direct 0    0            192.168.248.1   Vlan250
192.168.248.0/32    Direct 0    0            192.168.248.1   Vlan250
192.168.248.1/32    Direct 0    0            127.0.0.1       InLoop0
192.168.248.255/32  Direct 0    0            192.168.248.1   Vlan250
224.0.0.0/4         Direct 0    0            0.0.0.0         NULL0
224.0.0.0/24        Direct 0    0            0.0.0.0         NULL0
255.255.255.255/32  Direct 0    0            127.0.0.1       InLoop0

thanks, I cannot see anything wrong.
if you plug in a laptop in the vlan 250, can you see in syslog if it attempts to get an IP ?

Yes. I even tried running dhclient to no avail:

host adelina has a few nic, right?

what does ip route show on that host? ifconfig

Since your switch is L3, do you have DHCP helper addresses setup on the SVI for the VLAN you’ve added? In the Cisco world, this is how they’re referenced. You need to be able to forward the DHCP broadcasts up to the actual server and this config does just that. Not sure what that setup looks like in HP/Aruba OS, but the concept should carry over.