Truenas Scale 25.04, Unifi, VLANs and routing

Networking & Firewalls
1

Hello folks,
first post and an avid fan of the Lawrence Systems YT channel.

TL:DR: I have too many NICs and VLANs on my TrueNAS server…

The long story is:
Yesterday evening I successfully setup a lot of TrueNAS vlan interfaces and bridges etc. And now it came down to my storrage vlan. I guess this is more of a best practice question, but let me lay down the setup real quick.

I have the following VLANs
VLAN10 - Servers
VLAN11 - Workstations
VLAN12 - IoT Devices

and the following NICs
eno1
eno2
eno3

I setup a vlan interface and bridge on eno 2 for a Syncthing app container for VLAN12 (to sync some things from phones and tablets) and I setup another vlan interface and bridge on eno 3 for Jellyfin for VLAN12.

Now I wanted to have my storage connection on eno1 in VLAN10 and VLAN11. And this is where my lack of knowledge showed regarding bridges. I thought they kind of act like a switch and I can have several vlan interfaces as members under the same bridge. Meaning tagged traffic comes into the bridge…and then the bridge routes and says“ah that’s VLAN10…there you go”. But I learned that I have to have a bridge per vlan interface and also a separate IP per vlan ON the bridge (and not the vlan interface).

So now I have two ways to go:

  1. Let my unifi switch do the routing between vlans. Means on the port on the switch that runs to eno1, I only allow traffic from vlan10 and vlan11 (native = none) and I don’t have vlan and bridges on TrueNAS at all.
  2. I create a vlan and bridge for each vlan id and every NIC on TrueNAS, set the tagged traffic on the switch port to the vlans.

Does that make sense? Did I get this right?

Advantage of 1 feels like it’s way easier to setup and maintain, because I only handle switch ports and vlans on the switch. But I also route every storage traffic through the switch and the firewall, correct?

Advantage of 2 is, that I have a little TrueNAS connection sitting in every VLAN and devices have storage access without much traffic on the firewall. But I will have about 5 pairs of vlan interfaces and bridges and additional ip addresses and all that mumbo jumbo stuff happening on TrueNAS.

Also my mind kind of blocks the fact that there is a NIC in truenas that has 3 bridges on it all with their own IP. How does that look like on the switch? Which device will be shown on my layer 2 USW-24-G2 unifi switch in the port view? I am baffled.

So what would you do? Go with option one and let the switch do all the lifting…or go with option two? And what happens if ….tadaaaa… I pull eno4 out of my NIC-hat and I want to bond it with eno1 for redundancy and capacity? More vlan interfaces and bridges?

Awful complicated stuff for my little brain ^^

Thanks for any advice and opinions and help.

1 Like
2

In my setups I prefer not to use a bridge, but have the UniFi switch port set to all coming into the TrueNAS create a VLAN as needed. If you are looking for more performance, such as dedicated storage network, then using a NIC on the TrueNAS just for storage and setting Native VLAN / Network to that storage network on the UniFi switch for TrueNAS and each other host that needs access to that storage is how I would set that up.

2 Likes
3

This is probably incorrect, or at the very least it’s just a limitation of truenas. You can definitely have multiple vlans or NICs plug into the same linux bridge. I do it on every system I touch.

What you’re trying to do sounds pretty simple and straight forward, but with that said I have never messed with truenas so I don’t know what it is capable of.

You prefer not to use the internal bridge? This would limit your layer-2 traffic to your NIC speed (and physical switch). Why wouldn’t you use the internal linux bridge for this, it is so much faster?

Also, the more performant follow-up suggestion would still be limited by the line speed. Why not avoid all this if the disk storage is “local” to the box? (which is my assumption to this post)

1 Like
4

What make you say the bridge is faster?

1 Like
5

A little experience will easily prove this. Just test it. Setup a linux bridge and send some traffic across it. This will change your recommendation immediately.

For example. My five year old workstation sends 39Gbps across its bridge, and my old rinky dink protecli router at home pushes 7Gbps through two bridges. (I’ve never bothered to test my servers) All the NICs in the above examples are only 2.5GB. So clearly sending traffic down the line would be the choke point long before the bridge taps out.

I looked at that link you provided and remembered that TrueNAS is migrating off BSD. And the BSD bridge is apparently not very good - I think it is not multi-threaded or something. Basically, it stinks. If the new linux based TrueNAS has the limitation the OP mentioned, then it stinks too for no good technical reason.

Just pick a linux distro and build your own truenas. It’s not very hard. And it’s fun.

1 Like
6

I ran some testing this morning on my little Protecli home router after thinking about it’s results and my configuration. This little box is slow and I’m doing a lot more than I should on it (RAM constrained), but I was curious how much routing was impacting my throughput.

The way I tested this box’s config yesterday was at layer 2 & 3. This morning I cloned a container and tested throughput on one single bridge (my second bridge in this case). This bumped the throughput by about 1.2Gbps. Which is interesting. That clearly shows the cost of packet filtering & routing on this little resource constrained box. Fancier Flowtable fw rules might improve efficiency and performance, but I kind of doubt it on this hardware.

With all that said, even on this pokey slow hardware the internal bridge is the way to go. I’ve heard you correctly argue, “don’t route storage”. That should probably extend to hardware switching as well - in most cases. Sometimes external storage is fundamentally required. But if you don’t need to physically separate your compute from your storage by way of cable, then don’t do that unnecessarily. Please correct me if I’m wrong. I’m happy to be wrong.

On a side note… I have long held the practice of moving my “core switch” up into my linux router. This has been my way of avoiding the router on a stick constraints. Saves a little money not buying a fancy core switch and the complexity of setting it up. And at home I simply remove the need for a switch altogether given the 4 ports is enough for my home (two ports for APs, one for my workstation). Critical feedback on this idea is welcome.

2 Likes
7

Ok, so I also fiddled with it a little bit more.
Turns out, it’s in fact possible to have a bridge with more than one vlan interface as member. And it also turned out you can also just have IPs on the vlan interface (as long as there is no bridge involved).

So I setup a vlan interface per NIC and have the single ports in the respective VLAN on the switch. Now every NIC sits in the correct VLAN and I can even assign Apps an IP address on the VLAN that I want them to live in.

Now I will start tinkering with adding bridges, house several VLANs in one bridge and have Apps the bridge IP assigned. I don’t think I will be able to avoid routing storage completely, given some constraints from TrueNAS.

And thanks liquidjoe for the insights. For now I think I will stick to TrueNAS. But my homelab journey has more or less just started…so I definitely put it on the project list to build a NAS storage from scratch.

1 Like
8

Keep in mind the argument “don’t route storage” only applies to physical cable-based routing. You can route storage internally (on the linux host) just fine. This axiom probably assumes router on a stick topology - arguably a very skinny stick.

You shouldn’t “switch storage” either unless you physically have to. On modern hardware both switching & routing storage will most likely be limited by the same thing, the physical line speed (or switch backplane). The internal switch on modern hardware will crush most physical setups without breaking a sweat.

All of this may just be academic to you if Truenas doesn’t have a button for these features. Maybe just keep this in the back of your mind if you want to keep learning.

1 Like