VLAN in Jails and VMS on TrueNAS-12.0-RELEASE

KpuCko · November 16, 2020, 6:23pm

Hello, I’m trying to create some Jails/Plugins and VMs on my FreeNAS server which has to part of another network segment (vlan)

Brief explanation of the interfaces of the server:

I have igb0 and igb1, they are part of bond interface, which is using LACP aggregation and it is set to trunk (I’m coming from Cisco’s world), with no vlan restrictions (open trunk)

I also have created two vlan interfaces, the first one is vlan30, which is used to host the FreeNAS management IP (as tagged vlan)
and another one, named vlan310, which should be the vlan which provide network connectivity for the guests (for example).

Regarding this link: How to setup VLANs within FreeNAS 11.3 | TrueNAS Community and based on my previous experience I know I have to create bridge interface and combine it with the vlan interface, that is exactly what I’ve did. There is a bridge named bridge310, and vlan310 is part of this bridge.

Just to ensure all about network subletting is fine (only for the test) I’ve set IP address on this bridge and I was able to ping it from my lan network, later on I removed the IP address, because I don’t needed it, and I know (regarding the link above) it shouldn’t have IP address set.

So far so good, except that fact when I try to create new Jail/Plugin I end up with broken DNS resolution, some errors like pkgs.freebsd.org cannot be resolved.

I can confirm that, when I start the Jail appropriate tun interface is created and the bridge became its parent. So from my point of view, all should be fine ;> but it is not.

Let me show you few pictures of the current config, and we can discuss what to test further.
Any advises are welcomed.

So let’s recap - If I set IP for this vlan network on the bridge interface (on the host level) I’m able to ping it, which means the 802.1q is working fine on the host OS, but when I try to pass this communication to VM or Jail I can’t. I’m attaching a screenshot for the VM config, because I assume VM network config will be much easier to understand.

KpuCko · November 23, 2020, 1:09pm

Ok, I managed to get it working, beside the fact guests (no matter I use Jails or VM’s) are unable to get their IP address via DHCP, but if I set static IP everything works like charm.
Do I missed something?

kevdog · November 23, 2020, 7:30pm

I’m not sure why things aren’t working for you since I’m not having any problem but I don’t have LACP involved however that doesn’t seem to be the problem.

So I just want to confirm within TrueNAS you have your appropriate bridges and vlans paired. I have bridge0/vlan1, bridge20/vlan20, bridge30/vlan30, bridge40/vlan40. That’s how I have my pairs set up. I have everything entering freenas as Tagged traffic, so by default I don’t have any packets entering freenas as untagged. I’m not sure this is a necessity, however I just remember playing around with VLANs a long time ago and after hours of trial an error – tagging all traffic for me was what worked.

With jail setup, I created a test jail named ThrowAway. Before I brought the jail up, I ran the following command to create to NICs within the jail, with each NIC being on a separate network – information taken from this post (SOLVED - Iocage Jails Multiple Interfaces? (VNET) | TrueNAS Community):

iocage set ip4_addr="vnet0|10.0.1.0/24,vnet1|10.0.40.0/24" ThrowAway

I then went went to configure or Edit the Jail settings. Under Basic Properties, I had DHCP Autoconfigure IPv4, VNET, and Berkeley Packet Filter checked. (Make sure you save the configuration at this point and the re-enter the settings to make more changes). Under the Network Properties I set the interfaces line as the following:

vnet0:bridge0,vnet1:bridge40

I save the settings again and then started the jail. I shelled into jail and had the following result:

Both interfaces were pingable from a remote computer.

I’m not sure if that helps you.

KpuCko · November 24, 2020, 10:24am

Thanks for the replay @kevdog

Let me show you how to the things looks in my side:

sofx-nas01# ifconfig
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: connected to UBNT (Port 6)
    options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 3c:ec:ef:20:51:74
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: connected to UBNT (Port 7)
    options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 3c:ec:ef:20:51:74
    hwaddr 3c:ec:ef:20:51:75
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
    groups: pflog
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: lagg0
    options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 3c:ec:ef:20:51:74
    laggproto lacp lagghash l2,l3,l4
    laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    groups: lagg
    media: Ethernet autoselect
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
vlan310: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vlan310 (192.168.10.0/24)
    options=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 3c:ec:ef:20:51:74
    groups: vlan
    vlan: 310 vlanpcp: 0 parent interface: lagg0
    media: Ethernet autoselect
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
vlan30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vlan30 (192.168.0.0/24)
    options=200401<RXCSUM,LRO,RXCSUM_IPV6>
    ether 3c:ec:ef:20:51:74
    groups: vlan
    vlan: 30 vlanpcp: 0 parent interface: lagg0
    media: Ethernet autoselect
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
vlan320: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vlan320 (192.168.20.0/24)
    options=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 3c:ec:ef:20:51:74
    groups: vlan
    vlan: 320 vlanpcp: 0 parent interface: lagg0
    media: Ethernet autoselect
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
bridge310: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Vlan310 (bridge)
    ether 02:9d:bf:09:66:36
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vlan310 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 2000000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
bridge30: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Vlan30 (bridge)
    ether 02:9d:bf:09:66:1e
    inet 192.168.0.12 netmask 0xffffff00 broadcast 192.168.0.255
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 12 priority 128 path cost 2000000
    member: vnet0.10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 13 priority 128 path cost 2000
    member: vlan30 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 7 priority 128 path cost 2000000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
bridge320: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Vlan320 (bridge)
    ether 02:9d:bf:09:66:40
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vlan320 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 8 priority 128 path cost 2000000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.10: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: sofxunifi01 as nic: epair0b
    options=8<VLAN_MTU>
    ether 3e:ec:ef:e6:1e:79
    hwaddr 02:8d:58:e6:f1:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>
vnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=80000<LINKSTATE>
    ether fe:a0:98:7a:7c:f1
    hwaddr 58:9c:fc:10:ff:c7
    groups: tap
    media: Ethernet autoselect
    status: active
    nd6 options=1<PERFORMNUD>
    Opened by PID 40694
sofx-nas01#

So as you see, the only one difference is that I’m not using vlan1 and bridge0.

But you can also check that, I have two guests, one VM and one Jail, and their interfaces are correctly assigned to the bridges (see the members)

Here is all the settings for the jail, output generated by iocage:

sofx-nas01# iocage get all sofxunifi01
CONFIG_VERSION:28
allow_chflags:0
allow_mlock:0
allow_mount:0
allow_mount_devfs:0
allow_mount_fusefs:0
allow_mount_nullfs:0
allow_mount_procfs:0
allow_mount_tmpfs:0
allow_mount_zfs:0
allow_quotas:0
allow_raw_sockets:0
allow_set_hostname:1
allow_socket_af:0
allow_sysvipc:0
allow_tun:0
allow_vmm:0
assign_localhost:0
available:readonly
basejail:0
boot:0
bpf:1
children_max:0
cloned_release:12.2-RELEASE
comment:Used to host Unifi Controller service
compression:lz4
compressratio:readonly
coredumpsize:off
count:1
cpuset:off
cputime:off
datasize:off
dedup:off
defaultrouter:192.168.0.1
defaultrouter6:auto
depends:none
devfs_ruleset:4
dhcp:0
enforce_statfs:2
exec_clean:1
exec_created:/usr/bin/true
exec_fib:0
exec_jail_user:root
exec_poststart:/usr/bin/true
exec_poststop:/usr/bin/true
exec_prestart:/usr/bin/true
exec_prestop:/usr/bin/true
exec_start:/bin/sh /etc/rc
exec_stop:/bin/sh /etc/rc.shutdown
exec_system_jail_user:0
exec_system_user:root
exec_timeout:60
host_domainname:home.lan
host_hostname:sofxunifi01
host_hostuuid:sofxunifi01
host_time:1
hostid:c42d3e00-e775-11e9-8000-3cecef205174
hostid_strict_check:0
interfaces:vnet0:bridge30
ip4:new
ip4_addr:vnet0|192.168.0.9/24
ip4_saddrsel:1
ip6:new
ip6_addr:none
ip6_saddrsel:1
ip_hostname:0
jail_zfs:0
jail_zfs_dataset:iocage/jails/sofxunifi01/data
jail_zfs_mountpoint:none
last_started:2020-11-20 18:26:37
localhost_ip:none
login_flags:-f root
mac_prefix:3eecef
maxproc:off
memorylocked:off
memoryuse:off
min_dyn_devfs_ruleset:1000
mount_devfs:1
mount_fdescfs:1
mount_linprocfs:0
mount_procfs:0
mountpoint:readonly
msgqqueued:off
msgqsize:off
nat:0
nat_backend:ipfw
nat_forwards:none
nat_interface:none
nat_prefix:172.16
nmsgq:off
notes:Used to host Unifi Controller service
nsem:off
nsemop:off
nshm:off
nthr:off
openfiles:off
origin:readonly
owner:root
pcpu:off
plugin_name:none
plugin_repository:none
priority:99
pseudoterminals:off
quota:none
readbps:off
readiops:off
release:12.2-RELEASE
reservation:none
resolver:nameserver 192.168.0.1
rlimits:off
rtsold:0
securelevel:2
shmsize:off
stacksize:off
state:up
stop_timeout:30
swapuse:off
sync_state:none
sync_target:none
sync_tgt_zpool:none
sysvmsg:new
sysvsem:new
sysvshm:new
template:0
type:jail
used:readonly
vmemoryuse:off
vnet:1
vnet0_mac:3eecefe61e79 3eecefe61e7a
vnet0_mtu:auto
vnet1_mac:none
vnet1_mtu:auto
vnet2_mac:none
vnet2_mtu:auto
vnet3_mac:none
vnet3_mtu:auto
vnet_default_interface:none
vnet_default_mtu:1500
vnet_interfaces:none
wallclock:off
writebps:off
writeiops:off
sofx-nas01#

Do you see any issues with the config?

KpuCko · November 24, 2020, 10:36am

By the way, I made a test, by shutting down the jail and set the IP address acquiring to DHCP, then tried to start the jail again, but look that:

KpuCko · December 5, 2020, 7:12am

Ok, issue solved.

The confusion came from forwarding mechanism. I left with the feeling that when I set net.inet.ip.forwarding to 1 on System -> Tunable as RC config, this will activate forwarding on boot. Unfortunately this is not true. So I have to set gateway_enable = Yes, and type RC to get forwarding enabled on boot.

With forwarding enabled, my guests VMs/Jails are able to get IP address by the DHCP service.
I’m sharing this with you and hope it will help to somebody.

kevdog · December 6, 2020, 5:13am

Weird, I don’t have any of those system tunables set. It’s probably my configuration is different than yours.