I’m aware this post is quite old, however I took this challenge on recently (possibly inspired by this post and was able to successfully setup Freenas Jails with VLAN). I took me about 3 days to work through all the problems, but I thought I’d post a solution for others since someone helped me with my problems (a detail of all my problems can be found here on this reddit thread: https://www.reddit.com/r/freenas/comments/eoe9vh/good_tutorial_for_setting_up_vlans_within_iocage/?sort=old
I’m sure there are many ways to solve this problem, however after reading through a bunch of posts,threads and other information gleaned from freenas forums, freebsd forums, reddit and the internet at large this is what worked:
First my setup:
pfsense router–>Unifi Switch–>Freenas (One cable entering main freenas installation on interface igb0, (other cable is for IPMI)).
I’m also using FreeNAS 11.2-U7 as the reference implementation combined with iocage jails. Changes to the FreeNAS system will either be though the GUI or through modifying system tunables.
There have also been some reports about VLAN/FreeNAS not working with some Intel Chipsets – (sorry I can’t find the thread right now on the FreeNAS forums however I’m not sure how widespread this issue is). I’m using the intel I210-AT found on most Supermicro Boards.
For purpose of this setup, I have default VLAN1 and VLAN30.
Based on recommendations from freebsd forums/freenas forums/reddit thread, they advised if running VLANs through FreeNAS not to mix untagged and tagged traffic. https://www.ixsystems.com/community/threads/vlan-setup-not-working.41303/ They suggested all traffic leaving switch to be tagged.
For unifi I set up a port profile (Settings->Profiles->Switch Ports). I called it TagAllNetworks and checked LAN and VLAN30 as members. I then applied this profile to the port connected to FreeNAS. Please note this TagAllNetworks profiles differs from the “All” profile created by Unifi in that it tags VLAN1 traffic whereas the default “All” profile leaves VLAN1 traffic untagged.
FreeNAS configuration
This is where things start to get a little tricky. In a nutshell you want to define all the VLANs inside the FreeNAS GUI and attach each VLAN to a different network bridge. You then want to selectively add jails to each bridge for each VLAN. So for example – create VLAN1/VLAN30 – VLAN1 to be associated with bridge0 and VLAN30 to be associated with bridge30. For your jails, add each jail to be a member of the correct VLAN by adding it to the bridge interace.
Pre-requisites – please be comfortable accessing your freenas via an alternative method (ie IPMI) prior to making changes to FreeNAS’ network stack. Most likely you will lose network connection to FreeNAS during setup and need to access FreeNAS through other means.
#1 Create VLANs
My Network->VLAN settings appeared as following:
VLAN Inteface Parent Interface VLAN TAG
vlan1 igb0 1
vlan30 igb0 30
#2 Modifiy Interfaces
This is where connectivity will likely be loss during settings. My original setting had only igb0 with associated IP address. You’ll need to delete all the interfaces contained in this section. After deleting the main interface you’ll need to obtain a shell into the freenas installation through the IPMI. Within the shell you can use a command such as:
ifconfig igb0 inet <IP address of FreeNAS> netmask 255.255.255.0
With the FreeNAS network brought up through the command line - you should be able to access the FreeNAS GUI in order to add Interfaces like the following:
Interface Name IPV4 Address Options
vlan1 vlan1 <IP Address of FreeNAS> up
vlan30 vlan30 up
#3 - Addition of System Tunables (This is what creates bridges and adds each VLAN to the appropriate bridge)
cloned_interfaces bridge0 bridge30 rc
ifconfig_bridge0 addm vlan1 up rc
ifconfig_bridge30 addm vlan30 up rc
ifconfig_igb0 up rc
(Note – if doing a lot of reading on this topic, I found I did not need to add parameters such as: net.add_addr, net.fibs, or set any routes)
–
Disable any jails or VMs that start at boot.
At this point I would reboot the system and examine the network interfaces at the command line.
You’ll want to ensure that the IP address of the FreeNAS installation is listed within the vlan1 section, and examine the bridge statements to ensure vlan1 is a member of bridge0 and vlan30 is a member of bridge30. Here is an example:
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 '^''^'mtu 1500
options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCS'^''^'UM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 0c:c4:7a:84:a5:94
hwaddr 0c:c4:7a:84:a5:94
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCS'^''^'UM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 0c:c4:7a:84:a5:95
hwaddr 0c:c4:7a:84:a5:95
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1'^'500
ether 02:f6:c7:64:02:00
nd6 options=9<PERFORMNUD,IFDISABLED>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 11 priority 128 path cost 2000000
member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 10 priority 128 path cost 2000
member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 9 priority 128 path cost 2000
member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 8 priority 128 path cost 2000
member: vlan1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 55
bridge30: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu '^'1500
ether 02:f6:c7:64:02:1e
nd6 options=9<PERFORMNUD,IFDISABLED>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vlan30 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 7 priority 128 path cost 55
vlan1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0'^' mtu 1500
options=200001<RXCSUM,RXCSUM_IPV6>
ether 0c:c4:7a:84:a5:94
inet 10.0.1.197 netmask 0xffffff00 broadcast 10.0.1.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 1 vlanpcp: 0 parent interface: igb0
groups: vlan
vlan30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric '^'0 mtu 1500
options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
ether 0c:c4:7a:84:a5:94
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 30 vlanpcp: 0 parent interface: igb0
groups: vlan
Examine any other bridges that might exist on the system. If any of the other bridges have the parent physical network card as a member – you’ll need to remove the interface from that particular bridge. For example – after a reboot I found this:
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1'^'500
ether 02:f6:c7:64:02:01
nd6 options=1<PERFORMNUD>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 20000
You do not want igb0 as member of this bridge. If you find you have this situation, either modify the process that established the bridge to remove igb0 as a member, or delete the igb0 member manually doing the following:
ifconfig bridge(X) deletem igb0
The reason you are deleting igb0 as member of other bridges is that all traffic entering FreeNAS is tagged and you need the tagged traffic to be handled by the appropriate network bridge. The appropriate bridge will untag the network traffic and then pass it untagged to its members. vlan1 should be the default interface rather than igb0.
#4. Jail Setup.
Briefly there is a lot of conflicting information on the internet how to setup the jails with VLANs inside of FreeNAS. This example is going to assume that the jails are setup using the freebsd VNET driver that provides each jail with a virtualized networking stack and allows for isolation of the jail traffic from being routed through the main FreeNAS installation. If looking at information on the web about VLANs/FreeNAS/Jails, you need to ascertain whether the author of the post is using or not using the VNET driver. This post on reddit explains difference of using between using and not using VNET: https://www.reddit.com/r/freenas/comments/9wp4a9/how_do_vlans_work_in_freenas_or_do_they/ea2gn97/. If you choose not to use the VNET approach, the following instructions listed below will not work, but could be modified. Assuming use of the VNET driver I configured each jail similar to:
VNET on
Berkley Packet Filter on
IPV4 Interface - vnet0
IPV4 Address - 10.0.1.156 #Option only needed if not using DHCP
IPV4 Netmask - 24 #Option only needed if not using DHCP
IPV4 Default Route - 10.0.1.1 #match this for your appropriate VLAN -- ie. VLAN1-10.0.1.1, VLAN30-10.0.30.1
vnet_interfaces none
interfaces - vnet0:bridge0 #match this for your appropriat VLAN -- ie VLAN1 - vnet0:bridge0, VLAN30 - vnet0:bridge30
exec_fib 0
resolver - /etc/resolv.conf
vnet_default_interface - auto
Please pay close attention to the interfaces line. This line associates each jails VNET network with the appropriate bridge. Modify the IPV4 address, subnet and gateway as appropriate for your particular VLAN if not using DHCP.
#5 VM setup
Because VLAN1 is now the default interface - you’ll need to ensure that each VM is associated with a particular VLAN network. For my VMs that were previously on the igb0 or untagged network, I modified the network settings so Virtual Machine->Devices->NIC->Nic to attach: VLAN1 (not igb0).
If you need you’re VMs on a different VLAN, specify the particular VLAN
That should be all that needs to be done to allow use of VLANs with FreeNAS. Might want to reboot and ensure everything is stable at this point.