FreeNAS Jails with VLAN

Not sure if this is networking or software :grinning:
Background:

  • I have VLANs on my pfSense firewall - working fine
  • I have my UniFi switch using the VLANs - working fine

I want to trunk VLANs into my FreeNAS jails.

I have the Unifi switch port set to all VLANs.
I have create a VLAN (numbered 101) for the 192.168.101.0/24 subnet.
I created the VLAN on FreeNAS on an unused NIC (igb3).
I created the jail with:

Basic Configuration

  • VNET - enabled
  • Berkely Packet Filter - enabled
  • IPv4 Interface - vnet0
  • IPv4 Address - 192.168.101.1
  • IPv4 Netmask - 24
  • IPv4 Default Router - 192.168.101.254

Network Properties

  • interfaces - vnet0:bridge101
  • vnet_default_interface - vlan101

I can ping the 192.168.101.254 gateway from FreeNAS (host).

This is likely something silly.

Any help/suggestions appreciated and welcome!

I tried to get something similar and had the same problem. In my case, it was allowing zoneminder in a jail to get access to my VLAN network for the IP cameras.

I was unable to get it to work within FreeNAS. I could see VLAN traffic on the host with tcpdump, but it wouldn’t pass through to the jail for whatever reason. You might try running tcpdump in the jail to check if traffic is getting through. I ended up using an additional network port on the server and a smart switch to filter and untag the VLAN traffic. Not ideal but it works.

Good luck.

Thanks @gzornetzer - No luck yet… Like you tcpdump sees the traffic from the jail but it cannot get out…

13:12:02.685398 00:25:90:0f:c4:1f (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 101, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has router.pelleys.com tell 192.168.101.1, length 28
13:12:03.748975 00:25:90:0f:c4:1f (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 101, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has router.pelleys.com tell 192.168.101.1, length 28

I’ve [posted to the FreeNAS forums - many looks but no answers. Here is some additional information (also from that thread):

I think I am getting closer. I forgot the tunables:
Added to variable: cloned_interfaces - value: vlan101 - type: rc
Added variable: ifconfig_bridge101 value: addm vlan101 up - type: rc
Here is the VLAN network configuration:
Vlan Interface: vlan101
Parent Interface: igb3
Vlan Tag: 101

Jail Basic Properties:
VNET: Selected
Berkley Packet Filter: Selected
IPv4 Interface: vnet0
IPv4 Address: 192.168.101.1
IPv4 Netmask: 24
IPv4 Default Router: 192.168.101.254

Network Properties:
interfaces: vnet0:bridge101
vnet_default_interface: vlan101

I’m not sure about my Unifi port configuration.
I have VLAN30 as the native VLAN - this is the network that my WiFi uses. Will be shared with VLAN101.
I have the port tagged with VLAN101. Should that be on the port?

I’m starting to think it could be the VLANs on the UniFi switch port. I’ve tried all VLANs, VLAN30 as native with VLAN101 as tagged, VLAN101 as native, etc. No luck.

I’m hoping that @LTS_Tom who has worked with pfSense and FreeNAS and UniFi can peek into the forums. (Since watching his YouTube videos got me into this mess :stuck_out_tongue:)

was that tcpdump snippet taken using something like ‘tcpdump -i vnet0’ in the jail?

What does something like ‘tcpdump -i igb3’ show on the main freenas host instance? or ‘tcpdump -i vlan101’? At least you should be able to double-check that the switch is forwarding the the vlan traffic properly.

If you only need a single network to service the jails, why not just forward it untagged from the switch to your igb3 network port?

Hi @gzornetzer - I’m at work now but the tcpdump was taken from the host (not the jail) with (I think…) “tcpdump -vv -i igb3 | grep 192.168.101”. I’ll check using vlan101 tonight.

The challenge is that where I have my network segmented between “core” network, server network, trusted wired host network and untrusted wireless guest network. I want to move the “Internet of Insecure Thnings” (Hopefully Tom Lawarence doesn’t have that copyrighted :smile:) onto their own subnet. This will be a mix of wired and wireless devices (e.g., xboxes, PS4, are wired; AC unit, etc. are wireless). The trusted wired/wireless clients both use Emby for DLNA services. I did have DLNA working across subnets using pimd on pfSense (IGMP Proxy is broken) but I don’t like “routing” IPMG across subnets.

I am staring to think my issue is on the switch port side of things. That said, I have tried the switch port tagged with ALL, VLAN 101 as the native, VLAN101 as tagged, etc. I still get " 1f (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 101, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has router.pelleys.com tell 192.168.101.1, length 28"

I have to put a [dot] in place of the period in the hostnames. Forum is picking it up as a link :thinking:

Here’s the dump using "tcpdump -vv -i vlan101 | grep 192.168.101:
18:09:58.030870 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has router[dot]pelleys[dot]com tell 192.168.101.1, length 28
18:09:59.081688 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has router[dot]pelleys[dot]com tell 192.168.101.1, length 28
18:10:00.145060 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has router[dot]pelley[dot].com tell 192.168.101.1, length 28

Comparison with igb0 capture:
18:14:15.139102 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has router[dot]pelleys[dot]com tell 192.168.101.1, length 28
18:14:16.184518 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has router[dot]pelleys[dot]com tell 192.168.101.1, length 28
18:14:17.248119 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has router[dot]pelleys[dot]com tell 192.168.101.1, length 28

For chuckles and giggles I did the same capture for igb0, igb1 and ibg2. No packets were captured.

I seems that part of the networking - at least on the host - is working. Could it be my switch VLAN tagging?

I just ran tcpdump in the jail. Here is the output:

18:19:59.723686 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.101.254 tell test101, length 28
18:20:00.768874 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.101.254 tell test101, length 28
18:20:01.832370 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.101.254 tell test101, length 28

Sorry for the radio silence - that tcpdump looks like just the arp requests from the host or the jail. I’m not seeing any traffic from other hosts.
For me, it wasn’t a problem for traffic to get passed between the jail and the host. The problem was that external traffic on the VLAN wasn’t reaching the jail. External VLAN traffic would reach the freenas host, though.
You might be right about it being a switch problem if you don’t see vlan traffic on the host.

Hi @gzornetzer - I’m really sure where the issue is. On the UniFi switch port side, I think that I have covered all of the possible combinations - untagged, Native VLAN30 w/ tagged VLAN101, native VLAN101 (turning off the jail that is using VLAN30 as that is not tagged - igb3 is acting as a simple physical port).

I am starting to think the issue may be with the physical prot as there has been some bugs reported about VLAN problems for FreeBSD (and also pfSense) where enabling “vlanhwtag” renders VLAN on i210/i350 not functional. I am not a FreeBSD (or FreeNAS, for that matter) expert so all help is greatly appreciated!

@nfld.republic - the bug report you found for pfsense is running in a VM with HW passthrough - that adds a ton of complexity and room for error. I can’t speak for the bsd bug report, though it’s on freebsd 12, rather than 11 (which is the base for current pfsense).
The other piece of debugging advice is to look at the MAC address table in your switch. I don’t use unifi (all Netgear here), but I assume that you can filter the mac address table by VLAN - make sure that the switch is acknowledging the HW mac of the freenas box on the relevant VLAN’s. If not, it’s likely either a switch misconfig or the freenas box isn’t sending out the tagged packets.

I’m aware this post is quite old, however I took this challenge on recently (possibly inspired by this post and was able to successfully setup Freenas Jails with VLAN). I took me about 3 days to work through all the problems, but I thought I’d post a solution for others since someone helped me with my problems (a detail of all my problems can be found here on this reddit thread: https://www.reddit.com/r/freenas/comments/eoe9vh/good_tutorial_for_setting_up_vlans_within_iocage/?sort=old

I’m sure there are many ways to solve this problem, however after reading through a bunch of posts,threads and other information gleaned from freenas forums, freebsd forums, reddit and the internet at large this is what worked:

First my setup:
pfsense router–>Unifi Switch–>Freenas (One cable entering main freenas installation on interface igb0, (other cable is for IPMI)).

I’m also using FreeNAS 11.2-U7 as the reference implementation combined with iocage jails. Changes to the FreeNAS system will either be though the GUI or through modifying system tunables.

There have also been some reports about VLAN/FreeNAS not working with some Intel Chipsets – (sorry I can’t find the thread right now on the FreeNAS forums however I’m not sure how widespread this issue is). I’m using the intel I210-AT found on most Supermicro Boards.

For purpose of this setup, I have default VLAN1 and VLAN30.

Based on recommendations from freebsd forums/freenas forums/reddit thread, they advised if running VLANs through FreeNAS not to mix untagged and tagged traffic. https://www.ixsystems.com/community/threads/vlan-setup-not-working.41303/ They suggested all traffic leaving switch to be tagged.

For unifi I set up a port profile (Settings->Profiles->Switch Ports). I called it TagAllNetworks and checked LAN and VLAN30 as members. I then applied this profile to the port connected to FreeNAS. Please note this TagAllNetworks profiles differs from the “All” profile created by Unifi in that it tags VLAN1 traffic whereas the default “All” profile leaves VLAN1 traffic untagged.

FreeNAS configuration
This is where things start to get a little tricky. In a nutshell you want to define all the VLANs inside the FreeNAS GUI and attach each VLAN to a different network bridge. You then want to selectively add jails to each bridge for each VLAN. So for example – create VLAN1/VLAN30 – VLAN1 to be associated with bridge0 and VLAN30 to be associated with bridge30. For your jails, add each jail to be a member of the correct VLAN by adding it to the bridge interace.

Pre-requisites – please be comfortable accessing your freenas via an alternative method (ie IPMI) prior to making changes to FreeNAS’ network stack. Most likely you will lose network connection to FreeNAS during setup and need to access FreeNAS through other means.

#1 Create VLANs
My Network->VLAN settings appeared as following:

VLAN Inteface     Parent Interface    VLAN  TAG
vlan1                   igb0              1
vlan30                  igb0             30

#2 Modifiy Interfaces
This is where connectivity will likely be loss during settings. My original setting had only igb0 with associated IP address. You’ll need to delete all the interfaces contained in this section. After deleting the main interface you’ll need to obtain a shell into the freenas installation through the IPMI. Within the shell you can use a command such as:

ifconfig igb0 inet <IP address of FreeNAS> netmask 255.255.255.0

With the FreeNAS network brought up through the command line - you should be able to access the FreeNAS GUI in order to add Interfaces like the following:

Interface     Name    IPV4 Address              Options
vlan1         vlan1    <IP Address of FreeNAS>     up
vlan30        vlan30                               up

#3 - Addition of System Tunables (This is what creates bridges and adds each VLAN to the appropriate bridge)

cloned_interfaces	bridge0 bridge30	rc
ifconfig_bridge0	addm vlan1 up	    rc
ifconfig_bridge30	addm vlan30 up	    rc
ifconfig_igb0	    up	                rc

(Note – if doing a lot of reading on this topic, I found I did not need to add parameters such as: net.add_addr, net.fibs, or set any routes)


Disable any jails or VMs that start at boot.

At this point I would reboot the system and examine the network interfaces at the command line.
You’ll want to ensure that the IP address of the FreeNAS installation is listed within the vlan1 section, and examine the bridge statements to ensure vlan1 is a member of bridge0 and vlan30 is a member of bridge30. Here is an example:

igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 '^''^'mtu 1500
  options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCS'^''^'UM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
  ether 0c:c4:7a:84:a5:94
  hwaddr 0c:c4:7a:84:a5:94
  nd6 options=9<PERFORMNUD,IFDISABLED>
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCS'^''^'UM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
  ether 0c:c4:7a:84:a5:95
  hwaddr 0c:c4:7a:84:a5:95
  nd6 options=9<PERFORMNUD,IFDISABLED>
  media: Ethernet autoselect
  status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
  options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
  inet6 ::1 prefixlen 128
  inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
  inet 127.0.0.1 netmask 0xff000000
  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
  groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1'^'500
  ether 02:f6:c7:64:02:00
  nd6 options=9<PERFORMNUD,IFDISABLED>
  groups: bridge
  id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
  maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
  root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
  member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
          ifmaxaddr 0 port 11 priority 128 path cost 2000000
  member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
          ifmaxaddr 0 port 10 priority 128 path cost 2000
  member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
          ifmaxaddr 0 port 9 priority 128 path cost 2000
  member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
          ifmaxaddr 0 port 8 priority 128 path cost 2000
  member: vlan1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
          ifmaxaddr 0 port 6 priority 128 path cost 55
bridge30: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu '^'1500
  ether 02:f6:c7:64:02:1e
  nd6 options=9<PERFORMNUD,IFDISABLED>
  groups: bridge
  id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
  maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
  root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
  member: vlan30 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
          ifmaxaddr 0 port 7 priority 128 path cost 55
vlan1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0'^' mtu 1500
  options=200001<RXCSUM,RXCSUM_IPV6>
  ether 0c:c4:7a:84:a5:94
  inet 10.0.1.197 netmask 0xffffff00 broadcast 10.0.1.255
  nd6 options=9<PERFORMNUD,IFDISABLED>
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
  vlan: 1 vlanpcp: 0 parent interface: igb0
  groups: vlan
vlan30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric '^'0 mtu 1500
  options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
  ether 0c:c4:7a:84:a5:94
  nd6 options=9<PERFORMNUD,IFDISABLED>
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
  vlan: 30 vlanpcp: 0 parent interface: igb0
  groups: vlan

Examine any other bridges that might exist on the system. If any of the other bridges have the parent physical network card as a member – you’ll need to remove the interface from that particular bridge. For example – after a reboot I found this:

bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1'^'500
  ether 02:f6:c7:64:02:01
  nd6 options=1<PERFORMNUD>
  groups: bridge
  id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
  maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
  root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
  member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
          ifmaxaddr 0 port 1 priority 128 path cost 20000

You do not want igb0 as member of this bridge. If you find you have this situation, either modify the process that established the bridge to remove igb0 as a member, or delete the igb0 member manually doing the following:

ifconfig bridge(X) deletem igb0

The reason you are deleting igb0 as member of other bridges is that all traffic entering FreeNAS is tagged and you need the tagged traffic to be handled by the appropriate network bridge. The appropriate bridge will untag the network traffic and then pass it untagged to its members. vlan1 should be the default interface rather than igb0.

#4. Jail Setup.
Briefly there is a lot of conflicting information on the internet how to setup the jails with VLANs inside of FreeNAS. This example is going to assume that the jails are setup using the freebsd VNET driver that provides each jail with a virtualized networking stack and allows for isolation of the jail traffic from being routed through the main FreeNAS installation. If looking at information on the web about VLANs/FreeNAS/Jails, you need to ascertain whether the author of the post is using or not using the VNET driver. This post on reddit explains difference of using between using and not using VNET: https://www.reddit.com/r/freenas/comments/9wp4a9/how_do_vlans_work_in_freenas_or_do_they/ea2gn97/. If you choose not to use the VNET approach, the following instructions listed below will not work, but could be modified. Assuming use of the VNET driver I configured each jail similar to:

VNET on
Berkley Packet Filter on
IPV4 Interface - vnet0
IPV4 Address - 10.0.1.156   #Option only needed if not using DHCP
IPV4 Netmask - 24               #Option only needed if not using DHCP
IPV4 Default Route - 10.0.1.1  #match this for your appropriate VLAN -- ie. VLAN1-10.0.1.1, VLAN30-10.0.30.1
vnet_interfaces none
interfaces - vnet0:bridge0   #match this for your appropriat VLAN -- ie VLAN1 - vnet0:bridge0, VLAN30 - vnet0:bridge30
exec_fib 0
resolver - /etc/resolv.conf
vnet_default_interface - auto

Please pay close attention to the interfaces line. This line associates each jails VNET network with the appropriate bridge. Modify the IPV4 address, subnet and gateway as appropriate for your particular VLAN if not using DHCP.

#5 VM setup
Because VLAN1 is now the default interface - you’ll need to ensure that each VM is associated with a particular VLAN network. For my VMs that were previously on the igb0 or untagged network, I modified the network settings so Virtual Machine->Devices->NIC->Nic to attach: VLAN1 (not igb0).
If you need you’re VMs on a different VLAN, specify the particular VLAN

That should be all that needs to be done to allow use of VLANs with FreeNAS. Might want to reboot and ensure everything is stable at this point.

1 Like

@kevdog - Kevin, that is a GREAT writeup! Didn’t work for me though - I had followed the same route. I’m starting to think the issue is with the i350 chipset. I have to pick up a card with another Intel chipset and try that out. Is there another chipset that people have been successful with (Broadcom?)?

Thanks!

Only issue I was able to find with this i350 chipset is listed here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230996

It has to do more with enabling “vlanhwtag”.

After re-reading your post about 10 times (my fault :frowning:) and reviewing the ifconfig I finally picked out (I also widened my terminal to 132 or so columns to see it) that I had my interface (igb3) “physically” attached to a bridge. Once I removed it, it worked :grinning:

— Adding a couple of notes (1 or more :grin:):

  1. It may not make a difference but under the jail’s network configuration, you can set the vnet_default_interface as the VLAN itself instead of auto.
  2. On the UniFi port config, as @kevdog notes, you apparently must have the VLANs you want to pass through as being tagged; you cannot have the natvie VLANs as being one of the VLANs you want to pass through; e.g., if you have VLAN1000 as, say, your management VLAN, that one can be the native VLAN. The other VLANs must be tagged (likely self-evident to those who are better at networking than I)
  3. There are some posts that reference changing the default MTU from 1500 to 1504 to accomodate VLANs - that is not required.

@nfld.republic

Thanks for the notes

Addressing your points:

  1. vnet_default_interface – yes you are correct – you can set it as the VLAN number or leave it at auto. I didn’t find it made much difference in either case.
  2. I’ve read its recommended not to mix tagged and untagged traffic – I’m not certain if this is a hard and fast rule or just a recommendation. Certainly the setup would be simplified if VLAN1 traffic did not need to be tagged. I was hoping others would either help me confirm or deny this “recommendation”
  3. Changing of the MTU is not needed. I’m not sure if this needed to be done in the past, however as of 2019/2020 the MTU does not need to be adjusted

I upgraded over the weeked from 11.2U7 to 11.3RC2. Unfortunately/fortunately??? they changed the Network and VLAN interfaces and the instructions I wrote above are not entirely applicable to the new release. If you want I can add instructions for the 11.3 release – and probably just best to put these instructions within a new thread as I can see the instructions are likely to get buried in this thread.