XCP, pfsense & VLANs

I have a VDSL router in bridged mode connected to an Intel NUC. The NUC is then also connected to an Netgear GS724T switch. A UniFi AP completes the hardware aspects.

The NUC (XCP host) is connected to port 1 of the switch with the UniFi AP-AC-Pro on port 24.

I then defined 5 VLANs on the switch.

In XCP a network is created for each VLAN. This seems to be the only way as to address frame tagging within pfsense via a trunk interface.

Each network (as created within XCP) is then presented and can be assigned as an interface in pfsense.

VLANs cannot be specified within pfsense

Everything works perfectly fine at this point should there be a rule allowing anything anywhere on each interface.

However, the moment I amend the “Allow ANY to ANYWHERE” rule by specifying a SOURCE or DESTINATION, this new rule gets skipped.

Inter VLAN traffic fails the moment an inter VLAN rules is created, yet does traverse fine should a “Allow ANY to ANYWHERE” rule be specified.

My head truly hurts on how I’ve banged it continuously against this issue. Any guidance would be greatly appreciated.

NetGear GS724T switch config:

TLS-01

XCP’s network’s config:

PFsense’s interface assignments:

No option to specify VLANs within PFsense:

Example of traffic skipping intended VPN rule:

Have the same switch, I do not use the default lan for anything and start my vlans from 10 upwards for what it’s worth.

Given that you can define your vlans in pfsense independently of the switch, I’d say you have a config error in pfsense if you cannot add a vlan but you have interfaces.

@neogrid , is your pfsense also virtualised ?

No it’s on a cheap chinese box, however, I will eventually add PfSense to Proxmox but only for playing around.

In trying to understand the config issue better, how correct or incorrect is the following:

In this scenario, PFsense is basically a “router on a stick”. But instead of routing only IP traffic between virtual networks, it’s intention is to route tagged traffic on a physical switch.

As such, the single interface to the hypervisor has to be a VLAN trunk. However, the VLAN trunk configuration is required on the hypervisor (XCP) as to accept tagged packets into the virtual environment and also to tag packets correctly as they go onto the switch again.

From this point PFsense thus only needs to do the layer 3 IP routing. Noted that VLAN’s cannot be defined within PFsense once the sub-interfaces had been created within the hypervisor (XCP).

With each of the XCP sub interfaces (VLANs) then assigned (configured) as an interface to a particular network in PFsense, why would VLAN even need to be defined within PFsense ? For surely creating VLANs within PFsense would then result in creating sub-interface of the sub-interface. VLAN’s on VLAN’s. Or am I totally missing the point ?

Actually I first started trying to suss out pfSense in a vm, I wasted a lot of time.

However, it ought to be straight forward to get your setup working methinks. I use Proxmox, with that as a reference I can see how I would virtualise pfsense.

If I had two ports on my VM box one would be the WAN and the 2nd would be a trunk to the switch. Pfsense would be configured as desired, WAN, LAN, vLAN etc - so this is no different to having pfsense on a physical box so far. Now if I had a second VM running on vlan 10 (in proxmox) I would need to bridge the VM ethernet port to the physical port on the box.

With the above you can get to the WAN without any issues, however, if you want to access other devices on your physcial network you need a switch. The switch will have to mirror the vlans on pfsense.

“Noted that VLAN’s cannot be defined within PFsense once the sub-interfaces had been created within the hypervisor (XCP).”

not sure I follow this, I’m pretty sure you MUST define vlans in pfsense.

Perhaps where you have an error is in the config of the networking in XCP or the rules on the firewall.

Maybe what you can do is setup pfsense on a physical box and connect it to your switch, then get your inter-vlan traffic flowing. If your rules are ok then it will be the networking on XCP where the config error is.

https://xcp-ng.org/docs/guides.html#vlan-trunking-in-a-vm

This will tell you how to do vlans with xcp-ng and virtualized pfsense. I use method #1 and haven’t tried method #2.

@neogrid, I’m basically following method# 1 as noted by @kevdog which proves it do-able.

The first hurdle still being the xn driver of FreeBSD which does not support 802.1q tagging.

Traffic currently flows freely between all vLANs, VM’s and physical devices. Each vLAN has it’s own dedicated DHCP server which is serving each vLAN’s related devices without issues.

Which has me pondering my understanding of frame tagging:

  1. When a port is configured as U “untagged” - frames going into that port is tagged with the associated vLAN ID - thus placing that device on a specific vLAN. The tag is removed at the port ( U or T) the frame exits.
  2. When a port is configured as T “tagged” - only frames that are already tagged with the associated vLAN ID, is allowed into the port. The T relates to “Tagged” and NOT Trunk as newcomers would assume. The tag is removed at the port ( U or T ) the frame exits.
  3. When a port is not configured - All traffic, tagged or untagged is allowed either in or out of the port.

Am I missing the ball somewhere above ?

Yes I would agree on point 1, w.r.t. to netgear switches it looks to me you create a trunk by tagging the ports with the vlans, at least I do when I daisy-chain a second port. Point 3, I’m not sure I’ve not tested this. So basically we have the same understanding of the switch.

I’m not clear why you can’t add vlans in pfsense, you ought to be able to see your interfaces. What happens when you try to add a vlan in pfsense ?

The interfaces in PFsense is the virtual interfaces created within XCP of which the vLAN association is done upon creating the virtual interfaces in XCP. Matching the XCP interfaces in PFsense according to the “VLAN logic” to be applied in PFsense is important as the packet tagging and removal is done within XCP and not PFsense.

Make no mistake, this was quite a mind bender. As per : https://xcp-ng.org/docs/guides.html#vlan-trunking-in-a-vm - “the xn driver does not support 802.1q, pfSense will not allow you to create vlans on any interface using the xn driver.” Is there an alternate driver to date that fixes this ?

Regardless, after 4 months of tinkering with mixed results, I finally got a small win this morning at 4 adjusting the tagging on the switch … but now my Unifi Controller and AP isn’t playing nicely. And prior to defaulting all rules on each interface back to “Allow ANY - ANY” video streaming to the Chromecast was poor with slow response.

On the switch I have vLANs
10 - kids, cause, can you trust them to not bring malware into the system …
20 - the local lan and thus preventing any stray “connection” to have any access of any sorts
30 - VoiP, not used
40 - IoT, Google Mini & Chromecast
50 - Docker image running UniFi Controller

Port 1 connects to XCP
Port 23 connects to the Unifi AP

vLAN membership:
10 - Port 1, Port 23 - Tagged
20 - Port 1, Port 23 - Tagged
30 - Port 1, Port 23 - Tagged
40 - Port 1, Port 23 - Tagged
50 - Port 1 - Tagged , Port 23 - Untagged

Port 23 in Untagged as the management traffic of the Unifi AP needs to get to the UniFi Controller, being on vLAN 50.

The UniFi AP presents vLANs 10 (kids) , 20 (LAN) and 40 (IoT) as individual wLANs access points.

On PFsense each interface has the same “Allow ANY - ANY” rule.

Yet the Unifi AP is not seen by the Unifi Controller, although it does lease it’s IP off the 50 vLAN. I believe if I get behind this setback, all should hopefully fall in place.

that’s obviously the problem, i thought it was the standard for vlans. Sorry I don’t have a solution for that, however, others have virtualised pfsense and must have vlans running … they might have something to add …

Can you post some screenshots how you setup pfsense and with the Unifi controller?

Don’t worry about the xn driver not supporting VLANs. The methods described in the vlan trunking article work – I have a working setup with xcp-ng, pfsense, and Unifi controller with unifi switch. I use method #1 described in the link.

My only other thought is the docker image running the Unifi Controller. Are you sure you’ve set this part up correctly with the docker host.

SSH sessions between vLANs connect, but intermittently time-out for a few irritating seconds without closing the SSH session.

Being fixated on the switch, I validated that the latest version of the firmware is running on the switch.

Large downloads between vLANs were fine. With no ideas what to try, I then started stopping individual services on PFsense, whilst testing the SSH situation subsequently.

By sheer luck I noticed that the SSH sessions’ time-outs did not occur when I connected via IPv4.

Since I don’t actively use OpenVPN, with a preference for WireGuard the moment it becomes available, I uninstalled OpenVPN. For good measures, followed by a reboot.

Lo and behold, I have not had a SSH session which times-out within a session as yet, though it’s still early days.

@kevdog, I’ve posted the most relevant configs I relate to in the second post of the thread. Should you require anything more specific, please ask and I provide.

A smartphone with Google Home on vLAN20 can access the Google Mini and ChromeCast on vLAN40.

The same smartphone can connect via the Plex Android app to the Plex server on vLAN50 and list content, yet upon tapping play, the circle doesn’t stop turning whist the Plex eventually reports on screen (via the ChomeCast): “Sorry! Something went wrong”

Connecting the smartphone to vLAN40 and all is well, except that it circumvents the entire idea of segregating the IoT devices…

Herewith my current config:

The only aspect standing out here i.m.h.o is port 23 being marked “untagged” for vLAN50. This is specifically as to allow the Unifi AP’s management traffic to be tagged as the Unifi Controller is on vLAN50.

The following are all disabled:

  • Auto-Denial-of-Service
  • Green Ethernet

NTP is configured as to eliminate any form of timing issue.

On XCP the vLANs are created as understood from METHOD #1

The XCP intefaces (vLANs) is then assigned within PFsense. Interface HENETv6 is currently disabled as my ISP now supports IPv6 natively.

2020-05-26 PFsense-2

All services running on PFsense.

PFsense rules per vLAN.

Avahi setup as to allow Google Home on a smartphone (vLAN20) to reach the actual Google IoT devices on vLAN20

DHCPv4 is configured on each vLAN to provide that vLAN with it’s gateway and DNS, being the vLAN interface of PFsense.

DHCPv6 is not configured. RA (Router advertisement) is enabled, advising an IPv6 SLAAC interface of 2x IPv6 prefixes: i.e. fd10:: and a subnetwork of the ISP’s provided /56 prefix.

2020-05-26 Unifi AP

The Unifi Controller’s address is an IPv4 address within the Docker network of 172… I connect the AP to the controller via SSHing into the AP and executing the set-inform command.

NTP is configured on the Unifi Controller and the only other info configured being the association of vLANs 10, 20 and 40 each to a wLAN.

@kevdog, if you need any other information, please just say.

@Peek

I’ve looked at all the information you’ve posted – it’s definitely a lot to digest. Honestly however I’ve reread the entire thread and I’m a little lost at this point of what the problem is. Are you having a problem with mDNS and avahi? I’m asking this since you posted a problem with having a video play on your smartphone.

Third time lucky they say.

After the third complete redo from scratch, things seems to start working correctly…