@nfld.republic - the bug report you found for pfsense is running in a VM with HW passthrough - that adds a ton of complexity and room for error. I can’t speak for the bsd bug report, though it’s on freebsd 12, rather than 11 (which is the base for current pfsense).
The other piece of debugging advice is to look at the MAC address table in your switch. I don’t use unifi (all Netgear here), but I assume that you can filter the mac address table by VLAN - make sure that the switch is acknowledging the HW mac of the freenas box on the relevant VLAN’s. If not, it’s likely either a switch misconfig or the freenas box isn’t sending out the tagged packets.
I’m aware this post is quite old, however I took this challenge on recently (possibly inspired by this post and was able to successfully setup Freenas Jails with VLAN). I took me about 3 days to work through all the problems, but I thought I’d post a solution for others since someone helped me with my problems (a detail of all my problems can be found here on this reddit thread: https://www.reddit.com/r/freenas/comments/eoe9vh/good_tutorial_for_setting_up_vlans_within_iocage/?sort=old
I’m sure there are many ways to solve this problem, however after reading through a bunch of posts,threads and other information gleaned from freenas forums, freebsd forums, reddit and the internet at large this is what worked:
First my setup:
pfsense router–>Unifi Switch–>Freenas (One cable entering main freenas installation on interface igb0, (other cable is for IPMI)).
I’m also using FreeNAS 11.2-U7 as the reference implementation combined with iocage jails. Changes to the FreeNAS system will either be though the GUI or through modifying system tunables.
There have also been some reports about VLAN/FreeNAS not working with some Intel Chipsets – (sorry I can’t find the thread right now on the FreeNAS forums however I’m not sure how widespread this issue is). I’m using the intel I210-AT found on most Supermicro Boards.
For purpose of this setup, I have default VLAN1 and VLAN30.
Based on recommendations from freebsd forums/freenas forums/reddit thread, they advised if running VLANs through FreeNAS not to mix untagged and tagged traffic. https://www.ixsystems.com/community/threads/vlan-setup-not-working.41303/ They suggested all traffic leaving switch to be tagged.
For unifi I set up a port profile (Settings->Profiles->Switch Ports). I called it TagAllNetworks and checked LAN and VLAN30 as members. I then applied this profile to the port connected to FreeNAS. Please note this TagAllNetworks profiles differs from the “All” profile created by Unifi in that it tags VLAN1 traffic whereas the default “All” profile leaves VLAN1 traffic untagged.
FreeNAS configuration
This is where things start to get a little tricky. In a nutshell you want to define all the VLANs inside the FreeNAS GUI and attach each VLAN to a different network bridge. You then want to selectively add jails to each bridge for each VLAN. So for example – create VLAN1/VLAN30 – VLAN1 to be associated with bridge0 and VLAN30 to be associated with bridge30. For your jails, add each jail to be a member of the correct VLAN by adding it to the bridge interace.
Pre-requisites – please be comfortable accessing your freenas via an alternative method (ie IPMI) prior to making changes to FreeNAS’ network stack. Most likely you will lose network connection to FreeNAS during setup and need to access FreeNAS through other means.
#1 Create VLANs
My Network->VLAN settings appeared as following:
VLAN Inteface Parent Interface VLAN TAG
vlan1 igb0 1
vlan30 igb0 30
#2 Modifiy Interfaces
This is where connectivity will likely be loss during settings. My original setting had only igb0 with associated IP address. You’ll need to delete all the interfaces contained in this section. After deleting the main interface you’ll need to obtain a shell into the freenas installation through the IPMI. Within the shell you can use a command such as:
ifconfig igb0 inet <IP address of FreeNAS> netmask 255.255.255.0
With the FreeNAS network brought up through the command line - you should be able to access the FreeNAS GUI in order to add Interfaces like the following:
Interface Name IPV4 Address Options
vlan1 vlan1 <IP Address of FreeNAS> up
vlan30 vlan30 up
#3 - Addition of System Tunables (This is what creates bridges and adds each VLAN to the appropriate bridge)
cloned_interfaces bridge0 bridge30 rc
ifconfig_bridge0 addm vlan1 up rc
ifconfig_bridge30 addm vlan30 up rc
ifconfig_igb0 up rc
(Note – if doing a lot of reading on this topic, I found I did not need to add parameters such as: net.add_addr, net.fibs, or set any routes)
–
Disable any jails or VMs that start at boot.
At this point I would reboot the system and examine the network interfaces at the command line.
You’ll want to ensure that the IP address of the FreeNAS installation is listed within the vlan1 section, and examine the bridge statements to ensure vlan1 is a member of bridge0 and vlan30 is a member of bridge30. Here is an example:
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 '^''^'mtu 1500
options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCS'^''^'UM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 0c:c4:7a:84:a5:94
hwaddr 0c:c4:7a:84:a5:94
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCS'^''^'UM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 0c:c4:7a:84:a5:95
hwaddr 0c:c4:7a:84:a5:95
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1'^'500
ether 02:f6:c7:64:02:00
nd6 options=9<PERFORMNUD,IFDISABLED>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 11 priority 128 path cost 2000000
member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 10 priority 128 path cost 2000
member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 9 priority 128 path cost 2000
member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 8 priority 128 path cost 2000
member: vlan1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 55
bridge30: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu '^'1500
ether 02:f6:c7:64:02:1e
nd6 options=9<PERFORMNUD,IFDISABLED>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vlan30 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 7 priority 128 path cost 55
vlan1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0'^' mtu 1500
options=200001<RXCSUM,RXCSUM_IPV6>
ether 0c:c4:7a:84:a5:94
inet 10.0.1.197 netmask 0xffffff00 broadcast 10.0.1.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 1 vlanpcp: 0 parent interface: igb0
groups: vlan
vlan30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric '^'0 mtu 1500
options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
ether 0c:c4:7a:84:a5:94
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 30 vlanpcp: 0 parent interface: igb0
groups: vlan
Examine any other bridges that might exist on the system. If any of the other bridges have the parent physical network card as a member – you’ll need to remove the interface from that particular bridge. For example – after a reboot I found this:
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1'^'500
ether 02:f6:c7:64:02:01
nd6 options=1<PERFORMNUD>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 20000
You do not want igb0 as member of this bridge. If you find you have this situation, either modify the process that established the bridge to remove igb0 as a member, or delete the igb0 member manually doing the following:
ifconfig bridge(X) deletem igb0
The reason you are deleting igb0 as member of other bridges is that all traffic entering FreeNAS is tagged and you need the tagged traffic to be handled by the appropriate network bridge. The appropriate bridge will untag the network traffic and then pass it untagged to its members. vlan1 should be the default interface rather than igb0.
#4. Jail Setup.
Briefly there is a lot of conflicting information on the internet how to setup the jails with VLANs inside of FreeNAS. This example is going to assume that the jails are setup using the freebsd VNET driver that provides each jail with a virtualized networking stack and allows for isolation of the jail traffic from being routed through the main FreeNAS installation. If looking at information on the web about VLANs/FreeNAS/Jails, you need to ascertain whether the author of the post is using or not using the VNET driver. This post on reddit explains difference of using between using and not using VNET: https://www.reddit.com/r/freenas/comments/9wp4a9/how_do_vlans_work_in_freenas_or_do_they/ea2gn97/. If you choose not to use the VNET approach, the following instructions listed below will not work, but could be modified. Assuming use of the VNET driver I configured each jail similar to:
VNET on
Berkley Packet Filter on
IPV4 Interface - vnet0
IPV4 Address - 10.0.1.156 #Option only needed if not using DHCP
IPV4 Netmask - 24 #Option only needed if not using DHCP
IPV4 Default Route - 10.0.1.1 #match this for your appropriate VLAN -- ie. VLAN1-10.0.1.1, VLAN30-10.0.30.1
vnet_interfaces none
interfaces - vnet0:bridge0 #match this for your appropriat VLAN -- ie VLAN1 - vnet0:bridge0, VLAN30 - vnet0:bridge30
exec_fib 0
resolver - /etc/resolv.conf
vnet_default_interface - auto
Please pay close attention to the interfaces line. This line associates each jails VNET network with the appropriate bridge. Modify the IPV4 address, subnet and gateway as appropriate for your particular VLAN if not using DHCP.
#5 VM setup
Because VLAN1 is now the default interface - you’ll need to ensure that each VM is associated with a particular VLAN network. For my VMs that were previously on the igb0 or untagged network, I modified the network settings so Virtual Machine->Devices->NIC->Nic to attach: VLAN1 (not igb0).
If you need you’re VMs on a different VLAN, specify the particular VLAN
That should be all that needs to be done to allow use of VLANs with FreeNAS. Might want to reboot and ensure everything is stable at this point.
1 Like
@kevdog - Kevin, that is a GREAT writeup! Didn’t work for me though - I had followed the same route. I’m starting to think the issue is with the i350 chipset. I have to pick up a card with another Intel chipset and try that out. Is there another chipset that people have been successful with (Broadcom?)?
Thanks!
Only issue I was able to find with this i350 chipset is listed here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230996
It has to do more with enabling “vlanhwtag”.
After re-reading your post about 10 times (my fault 
) and reviewing the ifconfig I finally picked out (I also widened my terminal to 132 or so columns to see it) that I had my interface (igb3) “physically” attached to a bridge. Once I removed it, it worked 
— Adding a couple of notes (1 or more 
):
- It may not make a difference but under the jail’s network configuration, you can set the
vnet_default_interfaceas the VLAN itself instead of auto. - On the UniFi port config, as @kevdog notes, you apparently must have the VLANs you want to pass through as being tagged; you cannot have the natvie VLANs as being one of the VLANs you want to pass through; e.g., if you have VLAN1000 as, say, your management VLAN, that one can be the native VLAN. The other VLANs must be tagged (likely self-evident to those who are better at networking than I)
- There are some posts that reference changing the default MTU from 1500 to 1504 to accomodate VLANs - that is not required.
Thanks for the notes
Addressing your points:
- vnet_default_interface – yes you are correct – you can set it as the VLAN number or leave it at auto. I didn’t find it made much difference in either case.
- I’ve read its recommended not to mix tagged and untagged traffic – I’m not certain if this is a hard and fast rule or just a recommendation. Certainly the setup would be simplified if VLAN1 traffic did not need to be tagged. I was hoping others would either help me confirm or deny this “recommendation”
- Changing of the MTU is not needed. I’m not sure if this needed to be done in the past, however as of 2019/2020 the MTU does not need to be adjusted
I upgraded over the weeked from 11.2U7 to 11.3RC2. Unfortunately/fortunately??? they changed the Network and VLAN interfaces and the instructions I wrote above are not entirely applicable to the new release. If you want I can add instructions for the 11.3 release – and probably just best to put these instructions within a new thread as I can see the instructions are likely to get buried in this thread.
Continuing the discussion from FreeNAS Jails with VLAN:
I was running 11.3, and was able to use Tom’s video on jail VLAN and it worked fine. Then I upgraded to 11.3 U1 and now the jail VLAN no longer works. A write up for 11.3 U1 would be very helpful at this point. Thanks.
There seems to be a couple of things going on:
- When upgrading, the migration of the VLAN does not work correctly or only partially. I had 3 of my 4 jail networking work from 11.2-U7 and 11.3. All the jails would come up but the fourth jail would not get any networking. As well, the old VLANs and bridges from 11.2-U7 would not show up in under Network | Interfaces. Only deleting the VLANs and Bridges, recreating them and then rebooting seems to fix the issue. I have a bug report entered on this one on the iXsystems Jira bug tracker.
- Even after updating from 11.3 to 11.3-U1 my networking will only work with the DHCP option. If I try to manually set the IP/subnet/gateway the networking will not work. I think that this is on trace (others have reported it) in 11.3-U2.
@nfld.republic
I’m sitting still on 11.3 and have not upgrade to U1.
Do you have a link to this bug?
It’s unfortunate if all the VLAN jails of mine would require DHCP rather than static IP address.
In terms of a writeup for 11.3 which @vince spoke about – what information are you needing to know? In the video where Tom referenced a VLAN post - I was the author of that original post.
Hi @kevdog - " 11.3 Networking issues with Plugins - only DHCP works" [NAS-105114]
https://jira.ixsystems.com/browse/NAS-105114
(Side note – I’ve seen you over on the Freenas forums --)
Thanks for link. So it looks like I’ll sit tight on just 11.3 and wait until 11.3 U2 upgrade. It sometimes amazes me how things during the development process breaks working configurations. Thanks for the heads up. Is this only with plugin jails? What about jails created that were not using plugins?
Kevdog,
I think others have stated the issue already. My issue is slightly different. After upgrading from 11.3 to 11.3U1 the secondary network (IX0 is primary network nic and igb0 is secondary network used for VLAN) that I was using for the VLAN is now stuck at up. I cannot force it to be down. Further, I cannot use it for a VLAN anymore (when i try to do Tom’s method of attaching it to a vlan the jail will not start). I had a spare nic (igb1) and used that and now it is working fine using Tom’s method of using a second nic for freenas VLANS. Something transpired in the upgrade, although it showed no errors, that made igb0 stuck at up (I have tried ifconfig to bring it down to no avail, if i use ifconfig to bring up, freenas says its up, it will go down then up). Thus, I have a spare nic to work around. However, I thought I would add my two sense.
Honestly I’ve never worked with multiple NICs on my FreeNAS (although I do have a spare igb1 – I’ve never utilized it). When you say igb0 is stuck at “up”, are there any VLANs associated with the network interface? Is this interface set within the GUI or did you delete it? Is it listed in the GUI? Have you actually tried deleting and then re-adding the interface in the GUI? Is “up” specified under the Options line?
For igb0 there is no vlan or bridge associated with it. After the 11.3u1 update, and finding the vlan not working in the jail I deleted both the vlan and the bridge. There is no cable attached to igb0. ix0 has the freenas ip address. igb0 has no ip address. In the gui igbo has no options set and nothing is checked. Yet, the gui reports it as being up. On the surface I cannot see any reason why it would be stuck at up. I have checked system tunables in the gui and nothing is set. It just seems igb0 is stuck at up. I can still use it for an ip in the same network as ix0, but using it for a vlan in a jail will not work. The jail will not start. If I set the vlan in the jail using dchp I get a dhcp error. If I set an ip in the vlan network i get a vnet error. (note these last two lines are when I have set a vlan and bridge attached to igb0). This is where I am at. Maybe the 11.3U2 update will fix this.
Can you delete igb0 in the GuI? You can always re-add it back.
@kevdog - It was affects both the plugins and “manually” created jails. I had a jail created manually created for my UniFi Controller and the old 11.2-U6-created version did not have the VLANs or brideges show up under the Network information.
No I cannot. I get an error that only virtual interfaces can be deleted.
You should be able to do it through the console (either IPMI or physcailly connected keyboard/monitor).
I tried to delete igb0 through the console. It did not work.
I’ve read its recommended not to mix tagged and untagged traffic – I’m not certain if this is a hard and fast rule or just a recommendation. Certainly the setup would be simplified if VLAN1 traffic did not need to be tagged. I was hoping others would either help me confirm or deny this “recommendation”
Hi guys, first post here. Just registered as this thread got my hopes up (not too high though) to get my config working.
So first I need to clarify this as I think this is - one of many - flaws I have:
on my pfSense I have 2 intel NIC on the mobo and 4 other on an intel 1000 pro extension card which makes plenty to have all my VLANs without tagging sitting directly on parent interfaces like so:
On my FeeNAS I have two interfaces igb0 as 192.168.22.111 and igb1 as 192.168.33.111
I do not use LAGG and would like to set up a ZoneMinder jail connecting to my CCTV VLAN 192.168.33.x/24 to save footage of the cams. Cams traffic goes through a 4-port PoE switch which is then uplinked to a Netgear managed switch dedicated only for the CCTV VLAN. The igb1 from the FreeNAS is connected to this switch as well and the switch is uplinked to the CCTV parent NIC on the Intel 1000 pro.
Before I dig any further in the settings I’d like to know if this would work or should I tag my VLAN traffic anyway?!?