Multiple LAN interfaces single DHCP range

[MattE]

I could use some ideas/suggestions with the last stage of my new network setup.

My pfsense box is setup with multiple WAN connections (Business class connection and residential connection), and multiple LAN connections (10G to house switch and 10G to AGG switch in garage).

As a note, I don’t have the AGG switch yet.

All of the ISP equipment and the majority of my network equipment will be in the garage. The switch in the house is directly connected to the pfsense box (LAN_House), and will connect to the AGG switch in the garage. All of my servers will also connect to the AGG switch, as well as a 48 port switch for the rest of the devices in the garage.

I route the traffic for my office via the Business Class (BC) gateway, and everything else over the residential (RES) gateway. I have several VLANs that are working well currently and I’d rather not give them up.

From what I’m reading and understanding, I have 3 possible ways to get both of the LAN connections working:

1. Bridge the connections
2. Use a different subnet on each interface and setup the DHCP server accordingly
3. Setup a LAGG? not sure about this one, I don’t think it would serve my purpose.

What I am attempting to do, is to get both LAN_House and LAN_Garage interfaces to play nicely and still be able to use my VLANs. which that knocks #1 out of the situation.

Using a different subnet, I’m not opposed to… I would consider this like the classic “Cisco 5 router network” I’d have to figure out the IPs/subnets for each interface and the DHCP server, but then it would be a matter of setting up a static route between the 2 LAN connections, and then setup the VLANS on the other side, correct? I’m not sure on the VLAN aspect if that would work.

And honestly with the LAGG, I don’t think this will get me what I’m going for.

Please let me know if I’m missing something, or if there is a simpler/better option that would allow 2 LAN interfaces to either share an IP range or just act as one interface where VLANs can be used.

Thanks!

[brwainer]

Converting the LAN interface to a bridge and then adding the other LAN interface is your best path. It is going to be somewhat painful to do this after having set everything up. However, I would question whether you are ever going to need 2x10Gbps on the LAN side, and considering that traffic between the two LAN ports will incur some CPU load in bridging, and suggest instead that you only have one LAN port on the router (to whichever switch is right next to it) and the switches connected.

[Greg_E]

PFsense to garage switch, garage switch to house switch. With modern switches pfsense (wan connection) will be your bottle neck, not a switch to switch connection. A single hop is not going to impact much of anything, not even real time video like NDI or SRT.

[MattE]

I wasn’t really concerned about latency or link saturation, as there’s not that much in the house that would be able to do that, if anything. I just wanted to isolate traffic a bit, as there will be a fair amount of streaming happening from my media server and backup traffic to my NAS. Which I figured it would be nice to have it set for its own port/link. Eventually there will be some camera feeds from the house… that would probably be the more demanding of stuff.

I’m not necessarily opposed to just having one link between the house and the garage, I just figured it would be a good “future proofing” as much as one can anyway, and would provide some division of traffic that didn’t just need to go strait out to the internet.

As a frame of reference, here’s a higher level idea of where my brain was:

[brwainer]

Wait…. You were going to intentionally introduce a switch loop? Just… don’t. I don’t have the energy right now to explain why. Only have one of those fiber connections to the house.

Also, unless they’re in different VLANs, don’t have more than one connection on your servers (the ones that show both a DAC and a LAG). It will be a troubleshooting nightmare. If they are in different VLANs, only configure a default gateway on one interface.

[MattE]

Introducing a loop is not necessarily terrible… yes its not perfect for everything and can cause issues, but that is the point of Spanning Tree (or Rapid Spanning Tree), but I understand where you’re coming from.

The servers will not have more than one active connection per VLAN/subnet the 1Gb connections will be on a different subnet/VLAN (than the DAC), and in most cases would be a LAGG on the switch. Their current config is via the 1G connections, the 10G connections have not been setup yet.

[brwainer]

RSTP would take effect if you bridge the interfaces on the PFSense and enable RSTP on the bridge, but it seems like an unneeded bother to create the redundant connection. And to be clear, I work with enterprise and branch networks where RSTP is essential, and I just advocated for using RSTP to choose between two possible uplinks between one HQ campus building and another (the current Active-Active BGP setup is suspected to be causing issues). I advocate for making the solution the minimum complexity needed.

[MattE]

The more I think about things… I tend to agree (regarding complexity).

Question… I if I do a different subnet on the garage interface and create the same VLANs on that interface, as long as the ID’s match, would the unifi settings for the network still work?

Example:
I have a “office” network that I have set for VLAN 50 on the House LAN interface. I have associated firewall rules and a specific gateway that the traffic gets routed to. Eventually my office will be in the garage, but I will have one or 2 devices that go outside of it. It also is setup on WAPs via the unifi network application. If I use the same VLAN ID, on the garage subnet will the unifi networks stay happy? I will have at least one AP in the garage.

Things in this scenario I would have to do:

• I’ll have to create new VLANs (matching with the garage interface)
• I’ll have to duplicate the firewall rules for each VLAN (which won’t be too bad) on the garage interface.
• I’ll have to ensure that one subnet is not blocked from the other (which I think is default).

I would think that would be okay, any thoughts?

Maybe I’m just over-complicating or over-thinking

[brwainer]

I don’t see what added benefit this solution gives you, and considering the connection between the switches, you’re just going to end up with both subnets existing in the same VLAN and having a muddied broadcast/multicast domain. Like, anything using DHCP would get responses from both router interfaces.

[MattE]

Okay, I think I am finally dusting off enough of the cobwebs… I’ve been buried too long in other functions to actually network properly anymore, which I guess is why I’m asking all the questions.

If I were to make the link from the House switch to the AGG switch a non-active port (redundant), then setup the subnets as I described previously… would that work as I was hoping?

[brwainer]

I really am unsure what you actually want, and why you’re trying to do a bunch of work to implement things more complicated than they need to be.

-single 10gb connection from the router to a switch
-other switch(es) connect to that one - this is literally what “Aggregation” in USW-Agg means
-each VLAN ID is assigned 1:1 to a subnet - don’t reuse within your area of control
-All ports between a router or switch to another switch or AP are allowed to have all VLANs
-policing of VLAN membership is done at edge ports or by APs
-VLANs/subnets are defined based on use case / need, not physical location

[Jossk]

why do you have a the Pro 24 connected to your router and your Switch Aggregation?

I would remove the connection to the router. As for the servers, I have no experience with either Proxmox or ESXi (and very limited with Xen Orchestra, as I’m still learning), so I can’t really comment on the particularities of either, nor can I comment on HA setups (that’s another thing on my list to learn).

From my experience, in a SOHO setup, redundancy is unnecessary. If I were to keep all of the network equipment, I would have Router->Switch Aggregator->24 & 48 port switches, & servers to 48-port switch.

Or is it setup because you have certain VMs linked to a specific interface? If that is the case, I have encountered something similar in Portainer. It seems (I haven’t yet found anything to confirm or disprove), only one macvlan can be linked to a single port. So to have 2 macvlan, like I was planning, I would need 2 Ethernet ports on my server (details on another post).

If I have any info incorrect, please correct me.

[MattE]

What I originally wanted was to have a semi-redundant path mostly so that there was not a single point of failure in case of either hardware/physical or some other software issue… mostly because I always was taught to not cascade things as it makes the entire system rely on a single point of failure.

Now, all of that being said, is it absolutely necessary, no. I wanted to do that, but I realize that I was making things more complicated, and potentially creating issues I don’t need. No problem, and I appreciate the feedback from someone who has been doing it every day.

The reason I asked the last question is that I don’t currently have the aggregation switch, but I also don’t have the 48 port switch either, I forgot to mention that, and my apologies. Currently I only have a 1gb link between the garage lan interface and an older netgear prosafe switch, this was to ensure that the interface was functioning, and to bring maybe a few services back up that I am REALLY missing right now.

As far as the VLANs go, that’s exactly how I have them setup, and I agree completely regarding use case, that is how I use them. There are several other VLANs I have that work fine and that I didn’t mention because they don’t play into anything specific for this use case (they would have similar needs).

[MattE]

@Jossk To address the first question… well this is what we were discussing… and I will be removing the link to the router and only keeping the link to the AGG switch.

As far as the VMs go, there are some services/machines that get linked to specific interfaces (or will be), but most traffic will be over the 10gb DAC. The reason for at least most of the 1gb cabling is in case of failure, or if I need to give a VM a dedicated NIC. Sometimes management (not IPMI, those are seperate).

You are pretty much spot on though.

[Jossk]

Now that I’m on the same page as all of you.

Depending on what you are using your work VM(s) for, 1Gbe should be good enough.

If it were me, I’d just get the 48-port switch, & skip the AGG switch. Then again, I don’t have the budget (nor do I even have a rack to put it), so I would try to utilize whatever hardware I get to the max, before I start adding more. But that’s just me.

This is how I would set it up, so feel free to disagree and tell me as such.

The UI Pro switches have 4x SFP+ ports.
SFP+ ports on the 48-port:

1. → Router
2. → 24-Port Switch
3. → Freyr
4. → Yggdrasil

I don’t think having the NVR use RJ45 instead of SFP should make a noticeable difference (I could be wrong). Alternatively, if you really want to use SFP, you could use a multimode cable to the UI Pro 24-port.

[MattE]

@Jossk
The Plex server on Freyr and some of the other VMs would benefit greatly from 10g, but that’s about it for that machine, Yggdrasil is basically my NAS and backup machine that would also benefit quite a lot from a 10g connection… though I may throw a little more at it once I have the RAM, not sure yet, but for now its doing well.

I’m not sure about the purchase timing, a lot depends on other variables… I’m planning on having a couple of other servers connected in the relative near future, which is why I am getting the AGG switch sooner rather than later (these would be machines for my job). But yes, just the 48 port pro switch would be enough for my immediate needs… and who knows when I can get the NVR, it will happen eventually.

As far as the cabling goes for SFP, DAC is by far the most power efficient… which means generally less heat. MM SR optics being the next best thing, and lastly… RJ45, those converters get REALLY hot. I rather like the DACs as they keep things simple.

[pjdouillard]

OP, from what you wrote and after seeing your diagram, I think there are some network notions that you are missing. I think what you want to do with your network should be more in line like this:

I think your x.x.100.3 switch would be your core switch and from there you will network all your other switch.
In your diagram, you added a lot of network loops for redundancy (aka adding links between access switch) that instead should be done via LAGG between your core switch and your access switch.

Right now, with the info you gave us, I see a flat network and the only routing will be performed for connections going to the Internet. Everything else is on the same network. Now this could be a bad thing or a good thing depending on your needs.

Good things: if you need things really really really simple but not really secure - (novice network)
Bad things: you are mixing your management and your data and users, etc all on the same network. One hickup from either of these could wreck havock on everything else.

The recommendation would be to segreggate your network into VLANs for each type of service you have: management, users, servers, camera, wifi, etc. All those vlans would be routed and policed by your pfsense. The trunk you see between the core and pfsense is for that. And I would really suggest that you manage your inter-vlan traffic through pfsense (where you have full control and protection) instead of from your core switch (where you have zero control and protection).

Good luck!

[Jossk]

Good points @pjdouillard. Another thing I would like to point out, since you mentioned just using LAGG for redundancy, is that now the LAGG switch becomes a single point of failure. The solution to that, would be to add a 2nd LAGG switch in parallel.

Note: The connection between the pfS router and the LAGG switch is also a single point of failure.

[pjdouillard]

@Jossk that’s why in my drawing I wrote “stack ??” so your LAGG link can be from 2 (or more) switches on the stack. If you add a switch in parallel, you will need another subnet/vlans in order to avoid layer-2 loops. Stacking switches is the easiest.

[Jossk]

Fair enough. I must have missed that.