FreeNAS Jails with VLAN

Hi @gzornetzer - I’m at work now but the tcpdump was taken from the host (not the jail) with (I think…) “tcpdump -vv -i igb3 | grep 192.168.101”. I’ll check using vlan101 tonight.

The challenge is that where I have my network segmented between “core” network, server network, trusted wired host network and untrusted wireless guest network. I want to move the “Internet of Insecure Thnings” (Hopefully Tom Lawarence doesn’t have that copyrighted :smile:) onto their own subnet. This will be a mix of wired and wireless devices (e.g., xboxes, PS4, are wired; AC unit, etc. are wireless). The trusted wired/wireless clients both use Emby for DLNA services. I did have DLNA working across subnets using pimd on pfSense (IGMP Proxy is broken) but I don’t like “routing” IPMG across subnets.

I am staring to think my issue is on the switch port side of things. That said, I have tried the switch port tagged with ALL, VLAN 101 as the native, VLAN101 as tagged, etc. I still get " 1f (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 101, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has tell, length 28"

I have to put a [dot] in place of the period in the hostnames. Forum is picking it up as a link :thinking:

Here’s the dump using "tcpdump -vv -i vlan101 | grep 192.168.101:
18:09:58.030870 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has router[dot]pelleys[dot]com tell, length 28
18:09:59.081688 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has router[dot]pelleys[dot]com tell, length 28
18:10:00.145060 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has router[dot]pelley[dot].com tell, length 28

Comparison with igb0 capture:
18:14:15.139102 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has router[dot]pelleys[dot]com tell, length 28
18:14:16.184518 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has router[dot]pelleys[dot]com tell, length 28
18:14:17.248119 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has router[dot]pelleys[dot]com tell, length 28

For chuckles and giggles I did the same capture for igb0, igb1 and ibg2. No packets were captured.

I seems that part of the networking - at least on the host - is working. Could it be my switch VLAN tagging?

I just ran tcpdump in the jail. Here is the output:

18:19:59.723686 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has tell test101, length 28
18:20:00.768874 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has tell test101, length 28
18:20:01.832370 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has tell test101, length 28

Sorry for the radio silence - that tcpdump looks like just the arp requests from the host or the jail. I’m not seeing any traffic from other hosts.
For me, it wasn’t a problem for traffic to get passed between the jail and the host. The problem was that external traffic on the VLAN wasn’t reaching the jail. External VLAN traffic would reach the freenas host, though.
You might be right about it being a switch problem if you don’t see vlan traffic on the host.

Hi @gzornetzer - I’m really sure where the issue is. On the UniFi switch port side, I think that I have covered all of the possible combinations - untagged, Native VLAN30 w/ tagged VLAN101, native VLAN101 (turning off the jail that is using VLAN30 as that is not tagged - igb3 is acting as a simple physical port).

I am starting to think the issue may be with the physical prot as there has been some bugs reported about VLAN problems for FreeBSD (and also pfSense) where enabling “vlanhwtag” renders VLAN on i210/i350 not functional. I am not a FreeBSD (or FreeNAS, for that matter) expert so all help is greatly appreciated!

@nfld.republic - the bug report you found for pfsense is running in a VM with HW passthrough - that adds a ton of complexity and room for error. I can’t speak for the bsd bug report, though it’s on freebsd 12, rather than 11 (which is the base for current pfsense).
The other piece of debugging advice is to look at the MAC address table in your switch. I don’t use unifi (all Netgear here), but I assume that you can filter the mac address table by VLAN - make sure that the switch is acknowledging the HW mac of the freenas box on the relevant VLAN’s. If not, it’s likely either a switch misconfig or the freenas box isn’t sending out the tagged packets.

I’m aware this post is quite old, however I took this challenge on recently (possibly inspired by this post and was able to successfully setup Freenas Jails with VLAN). I took me about 3 days to work through all the problems, but I thought I’d post a solution for others since someone helped me with my problems (a detail of all my problems can be found here on this reddit thread:

I’m sure there are many ways to solve this problem, however after reading through a bunch of posts,threads and other information gleaned from freenas forums, freebsd forums, reddit and the internet at large this is what worked:

First my setup:
pfsense router–>Unifi Switch–>Freenas (One cable entering main freenas installation on interface igb0, (other cable is for IPMI)).

I’m also using FreeNAS 11.2-U7 as the reference implementation combined with iocage jails. Changes to the FreeNAS system will either be though the GUI or through modifying system tunables.

There have also been some reports about VLAN/FreeNAS not working with some Intel Chipsets – (sorry I can’t find the thread right now on the FreeNAS forums however I’m not sure how widespread this issue is). I’m using the intel I210-AT found on most Supermicro Boards.

For purpose of this setup, I have default VLAN1 and VLAN30.

Based on recommendations from freebsd forums/freenas forums/reddit thread, they advised if running VLANs through FreeNAS not to mix untagged and tagged traffic. They suggested all traffic leaving switch to be tagged.

For unifi I set up a port profile (Settings->Profiles->Switch Ports). I called it TagAllNetworks and checked LAN and VLAN30 as members. I then applied this profile to the port connected to FreeNAS. Please note this TagAllNetworks profiles differs from the “All” profile created by Unifi in that it tags VLAN1 traffic whereas the default “All” profile leaves VLAN1 traffic untagged.

FreeNAS configuration
This is where things start to get a little tricky. In a nutshell you want to define all the VLANs inside the FreeNAS GUI and attach each VLAN to a different network bridge. You then want to selectively add jails to each bridge for each VLAN. So for example – create VLAN1/VLAN30 – VLAN1 to be associated with bridge0 and VLAN30 to be associated with bridge30. For your jails, add each jail to be a member of the correct VLAN by adding it to the bridge interace.

Pre-requisites – please be comfortable accessing your freenas via an alternative method (ie IPMI) prior to making changes to FreeNAS’ network stack. Most likely you will lose network connection to FreeNAS during setup and need to access FreeNAS through other means.

#1 Create VLANs
My Network->VLAN settings appeared as following:

VLAN Inteface     Parent Interface    VLAN  TAG
vlan1                   igb0              1
vlan30                  igb0             30

#2 Modifiy Interfaces
This is where connectivity will likely be loss during settings. My original setting had only igb0 with associated IP address. You’ll need to delete all the interfaces contained in this section. After deleting the main interface you’ll need to obtain a shell into the freenas installation through the IPMI. Within the shell you can use a command such as:

ifconfig igb0 inet <IP address of FreeNAS> netmask

With the FreeNAS network brought up through the command line - you should be able to access the FreeNAS GUI in order to add Interfaces like the following:

Interface     Name    IPV4 Address              Options
vlan1         vlan1    <IP Address of FreeNAS>     up
vlan30        vlan30                               up

#3 - Addition of System Tunables (This is what creates bridges and adds each VLAN to the appropriate bridge)

cloned_interfaces	bridge0 bridge30	rc
ifconfig_bridge0	addm vlan1 up	    rc
ifconfig_bridge30	addm vlan30 up	    rc
ifconfig_igb0	    up	                rc

(Note – if doing a lot of reading on this topic, I found I did not need to add parameters such as: net.add_addr, net.fibs, or set any routes)

Disable any jails or VMs that start at boot.

At this point I would reboot the system and examine the network interfaces at the command line.
You’ll want to ensure that the IP address of the FreeNAS installation is listed within the vlan1 section, and examine the bridge statements to ensure vlan1 is a member of bridge0 and vlan30 is a member of bridge30. Here is an example:

igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 '^''^'mtu 1500
  ether 0c:c4:7a:84:a5:94
  hwaddr 0c:c4:7a:84:a5:94
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
  ether 0c:c4:7a:84:a5:95
  hwaddr 0c:c4:7a:84:a5:95
  media: Ethernet autoselect
  status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
  inet6 ::1 prefixlen 128
  inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
  inet netmask 0xff000000
  groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1'^'500
  ether 02:f6:c7:64:02:00
  groups: bridge
  id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
  maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
  root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
          ifmaxaddr 0 port 11 priority 128 path cost 2000000
  member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
          ifmaxaddr 0 port 10 priority 128 path cost 2000
  member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
          ifmaxaddr 0 port 9 priority 128 path cost 2000
  member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
          ifmaxaddr 0 port 8 priority 128 path cost 2000
          ifmaxaddr 0 port 6 priority 128 path cost 55
bridge30: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu '^'1500
  ether 02:f6:c7:64:02:1e
  groups: bridge
  id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
  maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
  root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
  member: vlan30 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
          ifmaxaddr 0 port 7 priority 128 path cost 55
vlan1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0'^' mtu 1500
  ether 0c:c4:7a:84:a5:94
  inet netmask 0xffffff00 broadcast
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
  vlan: 1 vlanpcp: 0 parent interface: igb0
  groups: vlan
vlan30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric '^'0 mtu 1500
  ether 0c:c4:7a:84:a5:94
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
  vlan: 30 vlanpcp: 0 parent interface: igb0
  groups: vlan

Examine any other bridges that might exist on the system. If any of the other bridges have the parent physical network card as a member – you’ll need to remove the interface from that particular bridge. For example – after a reboot I found this:

bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1'^'500
  ether 02:f6:c7:64:02:01
  nd6 options=1<PERFORMNUD>
  groups: bridge
  id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
  maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
  root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
          ifmaxaddr 0 port 1 priority 128 path cost 20000

You do not want igb0 as member of this bridge. If you find you have this situation, either modify the process that established the bridge to remove igb0 as a member, or delete the igb0 member manually doing the following:

ifconfig bridge(X) deletem igb0

The reason you are deleting igb0 as member of other bridges is that all traffic entering FreeNAS is tagged and you need the tagged traffic to be handled by the appropriate network bridge. The appropriate bridge will untag the network traffic and then pass it untagged to its members. vlan1 should be the default interface rather than igb0.

#4. Jail Setup.
Briefly there is a lot of conflicting information on the internet how to setup the jails with VLANs inside of FreeNAS. This example is going to assume that the jails are setup using the freebsd VNET driver that provides each jail with a virtualized networking stack and allows for isolation of the jail traffic from being routed through the main FreeNAS installation. If looking at information on the web about VLANs/FreeNAS/Jails, you need to ascertain whether the author of the post is using or not using the VNET driver. This post on reddit explains difference of using between using and not using VNET: If you choose not to use the VNET approach, the following instructions listed below will not work, but could be modified. Assuming use of the VNET driver I configured each jail similar to:

Berkley Packet Filter on
IPV4 Interface - vnet0
IPV4 Address -   #Option only needed if not using DHCP
IPV4 Netmask - 24               #Option only needed if not using DHCP
IPV4 Default Route -  #match this for your appropriate VLAN -- ie. VLAN1-, VLAN30-
vnet_interfaces none
interfaces - vnet0:bridge0   #match this for your appropriat VLAN -- ie VLAN1 - vnet0:bridge0, VLAN30 - vnet0:bridge30
exec_fib 0
resolver - /etc/resolv.conf
vnet_default_interface - auto

Please pay close attention to the interfaces line. This line associates each jails VNET network with the appropriate bridge. Modify the IPV4 address, subnet and gateway as appropriate for your particular VLAN if not using DHCP.

#5 VM setup
Because VLAN1 is now the default interface - you’ll need to ensure that each VM is associated with a particular VLAN network. For my VMs that were previously on the igb0 or untagged network, I modified the network settings so Virtual Machine->Devices->NIC->Nic to attach: VLAN1 (not igb0).
If you need you’re VMs on a different VLAN, specify the particular VLAN

That should be all that needs to be done to allow use of VLANs with FreeNAS. Might want to reboot and ensure everything is stable at this point.

1 Like

@kevdog - Kevin, that is a GREAT writeup! Didn’t work for me though - I had followed the same route. I’m starting to think the issue is with the i350 chipset. I have to pick up a card with another Intel chipset and try that out. Is there another chipset that people have been successful with (Broadcom?)?


Only issue I was able to find with this i350 chipset is listed here:

It has to do more with enabling “vlanhwtag”.

After re-reading your post about 10 times (my fault :frowning:) and reviewing the ifconfig I finally picked out (I also widened my terminal to 132 or so columns to see it) that I had my interface (igb3) “physically” attached to a bridge. Once I removed it, it worked :grinning:

— Adding a couple of notes (1 or more :grin:):

  1. It may not make a difference but under the jail’s network configuration, you can set the vnet_default_interface as the VLAN itself instead of auto.
  2. On the UniFi port config, as @kevdog notes, you apparently must have the VLANs you want to pass through as being tagged; you cannot have the natvie VLANs as being one of the VLANs you want to pass through; e.g., if you have VLAN1000 as, say, your management VLAN, that one can be the native VLAN. The other VLANs must be tagged (likely self-evident to those who are better at networking than I)
  3. There are some posts that reference changing the default MTU from 1500 to 1504 to accomodate VLANs - that is not required.


Thanks for the notes

Addressing your points:

  1. vnet_default_interface – yes you are correct – you can set it as the VLAN number or leave it at auto. I didn’t find it made much difference in either case.
  2. I’ve read its recommended not to mix tagged and untagged traffic – I’m not certain if this is a hard and fast rule or just a recommendation. Certainly the setup would be simplified if VLAN1 traffic did not need to be tagged. I was hoping others would either help me confirm or deny this “recommendation”
  3. Changing of the MTU is not needed. I’m not sure if this needed to be done in the past, however as of 2019/2020 the MTU does not need to be adjusted

I upgraded over the weeked from 11.2U7 to 11.3RC2. Unfortunately/fortunately??? they changed the Network and VLAN interfaces and the instructions I wrote above are not entirely applicable to the new release. If you want I can add instructions for the 11.3 release – and probably just best to put these instructions within a new thread as I can see the instructions are likely to get buried in this thread.

Continuing the discussion from FreeNAS Jails with VLAN:

I was running 11.3, and was able to use Tom’s video on jail VLAN and it worked fine. Then I upgraded to 11.3 U1 and now the jail VLAN no longer works. A write up for 11.3 U1 would be very helpful at this point. Thanks.

There seems to be a couple of things going on:

  1. When upgrading, the migration of the VLAN does not work correctly or only partially. I had 3 of my 4 jail networking work from 11.2-U7 and 11.3. All the jails would come up but the fourth jail would not get any networking. As well, the old VLANs and bridges from 11.2-U7 would not show up in under Network | Interfaces. Only deleting the VLANs and Bridges, recreating them and then rebooting seems to fix the issue. I have a bug report entered on this one on the iXsystems Jira bug tracker.
  2. Even after updating from 11.3 to 11.3-U1 my networking will only work with the DHCP option. If I try to manually set the IP/subnet/gateway the networking will not work. I think that this is on trace (others have reported it) in 11.3-U2.

I’m sitting still on 11.3 and have not upgrade to U1.
Do you have a link to this bug?

It’s unfortunate if all the VLAN jails of mine would require DHCP rather than static IP address.

In terms of a writeup for 11.3 which @vince spoke about – what information are you needing to know? In the video where Tom referenced a VLAN post - I was the author of that original post.

Hi @kevdog - " 11.3 Networking issues with Plugins - only DHCP works" [NAS-105114]


(Side note – I’ve seen you over on the Freenas forums --)

Thanks for link. So it looks like I’ll sit tight on just 11.3 and wait until 11.3 U2 upgrade. It sometimes amazes me how things during the development process breaks working configurations. Thanks for the heads up. Is this only with plugin jails? What about jails created that were not using plugins?

I think others have stated the issue already. My issue is slightly different. After upgrading from 11.3 to 11.3U1 the secondary network (IX0 is primary network nic and igb0 is secondary network used for VLAN) that I was using for the VLAN is now stuck at up. I cannot force it to be down. Further, I cannot use it for a VLAN anymore (when i try to do Tom’s method of attaching it to a vlan the jail will not start). I had a spare nic (igb1) and used that and now it is working fine using Tom’s method of using a second nic for freenas VLANS. Something transpired in the upgrade, although it showed no errors, that made igb0 stuck at up (I have tried ifconfig to bring it down to no avail, if i use ifconfig to bring up, freenas says its up, it will go down then up). Thus, I have a spare nic to work around. However, I thought I would add my two sense.


Honestly I’ve never worked with multiple NICs on my FreeNAS (although I do have a spare igb1 – I’ve never utilized it). When you say igb0 is stuck at “up”, are there any VLANs associated with the network interface? Is this interface set within the GUI or did you delete it? Is it listed in the GUI? Have you actually tried deleting and then re-adding the interface in the GUI? Is “up” specified under the Options line?

For igb0 there is no vlan or bridge associated with it. After the 11.3u1 update, and finding the vlan not working in the jail I deleted both the vlan and the bridge. There is no cable attached to igb0. ix0 has the freenas ip address. igb0 has no ip address. In the gui igbo has no options set and nothing is checked. Yet, the gui reports it as being up. On the surface I cannot see any reason why it would be stuck at up. I have checked system tunables in the gui and nothing is set. It just seems igb0 is stuck at up. I can still use it for an ip in the same network as ix0, but using it for a vlan in a jail will not work. The jail will not start. If I set the vlan in the jail using dchp I get a dhcp error. If I set an ip in the vlan network i get a vnet error. (note these last two lines are when I have set a vlan and bridge attached to igb0). This is where I am at. Maybe the 11.3U2 update will fix this.

Can you delete igb0 in the GuI? You can always re-add it back.