Hi guys,
I was hoping to crowdsource a little design knowledge for a good cause! My company (very small IT and industrial controls outfit) recently took out a lease on a 4 person office in new shared office / business centre in our town. The entire campus is all about incubating new business ventures, startups or even existing companies. You can hire as many or as little desks and offices as you need. The campus owners themselves are also a startup and are in there with everyone else! There is a great community spirit within it, and all tennants try to pass on work to other people in the building if its not something you can do yourself…
Anyways, in the interest of helping where I can, I have volunteered my time to improve and configure the IT infrastructure. The owner will pay for the hardware. The current set up is that we have 5 incoming 1G fibre connections. Each one has a fibre/copper ont, with a cat6 patch cables going to a domestic grade router. The 5 routers are spread aound the campus, with most people simply using the wifi connectivity. This was ok for a while, but as occupancy numbers rose, the craptastic routers fall over…
I’m totaly happy in the Unifi space, and can spec out and configure the layer 2 stuff and wifi APs etc. That said, I’ve never actually done a job with 5 WANs, so this is where I’d like some advice. I want to use a pfsense router(s), but I can’t find any with 5 physical wan ports. How would this typically be done? I could use 5 small pfsense appliances, but that sems silly and hard to manage for future admins…
I would prefer not to build a custom server with multiple nics, I was hoping to find some kind of off the shelf appliance or. I have used the sg1100 and 3100 models before and love them. They just work, and keep going forever!
What about connecting all the WAN connections (each on its own VLAN) to a switch that has at least one 10G port and connecting a router with a 10G capable NIC to that port? Obviously you need to make sure the switch can actually switch packets at at least 5Gbit/s and the router can route packages at that speed.
Yes, I never thought of it that way. This is like using a switch in reverse. Tag all the wans as vlans, trunk them all into pfsense, and declare 5 seperate gatewate interfaces to unbundle them back out and act like seperate connections (or at least that’s what I think you mean ).
I’ll do some research and see if I can find such a switch …
If anyone has done this before and already has a good make/model in mind let me know. If not I’ll share my findings back here in a few days…
Thanks for the suggestion!
That is precisely what I mean. For what it’s worth, I’m doing this at home with single WAN connection and only gigabit. I have my cable modem connected to a switch port configured as untagged vlan 99, my virtualized pfSense gets all vlans tagged on a single vNIC, then there is an interface within pfSense using that vNIC and vlan 99. Works perfectly. I don’t see why this couldn’t be done multiple times.
For the switch, if you don’t want a full 10G switch (perhaps because price is a concern), you might be able to find a managed gigabit switch with a 10G “uplink” port. Of course, it would really be a downlink in this case, but whatever, that’s how they get marketed…
Ok - So I’ve settled on a pfsense XG-7100 1U and a Netgear Switch XS708T, both will be linked together via sfp+.
However… following some on-site investigation, I discovered each of the incoming 5 WANs are already coming in as tagged VLANs, and they are all set to 10 by the ISP. It doesn’t matter right now as there are 5 separate routers (ISP supplied). I am not sure the proposed plan would work now, as on the potentially new aggregator switch I will have 5 incoming connection tagged as VLAN 10. I don’t think the switch can somehow remap the VLAN tags on the copper ports so that I would wind up with VLAN 10/20/30/40/50 coming out of the sfp+ uplink port…
Also, each of the current ISP provided routers is also sending PPPoE Login credentials to the ISP. I do know the creds, but unsure if PfSense would be able to pass them back correctly to the ISP in this proposed configuration…
PPPoE is not something I deal with much as it is not popular in America. I know pfsense supports it, but I know people have trouble with it. But I don’t know if those people did not RTFM, or if it is just difficult to work with.
Hi Tom, I have an SG-3100 to hand, I can bring it over there and try it out, on just one of the incoming fibres. I did read the docs, dont see why it wouldnt in theory. It was more the complication of doing it for 5 WAN connections at the same time on the one device - was wondering if anyone has tried that before…
Any tips or advice on how to get the 5 physical incoming connections hooked into PF sense? Each are already tagged VLAN 10 by the ISP, and PF Sense box only has 2 WAN ports…
I could be bordering on going beyond of the free forum help category here
I’m curious why the ISP uses PPPoE yet hands out VLANs, seems strange unless they’re taking a 5G Fibre line with a single IP and spreading it over 5 areas.
Adding to Tom’s comment on PPPoE, it hasn’t been widely used here in the states since the 90’s, early 00’s when DSL was a thing. I’d suppose there’s still area’s that use DSL (feels bad man) but it’s usage is small anymore.
Man, I did not think of that. I have not experienced an ISP tagging traffic myself, but I have heard of it. If you are still able to change out the switch (which I believe you mean is from Netgear, not Netgate) maybe you can get your hands on one that supports VLAN mapping / translation.
I found the FS S3900-24F4S which features 24 Gbit/s ports and 4 SFP+ ports. It has a switching capacity of 128 GBit/s and according to the manual, it supports VLAN translation. Disclaimer though: I don’t know anything more about this thing than is provided by the manufacturer’s website. I have never used FS switches.
Another thing I have been wondering about: If this is a campus and businesses can rent flexible office space, do you (or rather, campus management) consider renting dedicated internet connections to customers? Without knowing any more about that campus than you provided in your original question, I could imagine that sooner or later, (potential) customers that want their own IP (space) will show up. It sounds to me like right now, a client’s public IP changes depending on what router/wifi they are currently connected to which isn’t really ideal for on-premise hosting.
Hi, yes I did mean netgear, good catch!. Thanks for the tip on the switch, I’ll study that, and the topic of vlan translation in general. You are also bang on regarding future customers needing fixed ips internally within the building. Has not been a big problem yet, I got people by so far by forwarding a few ports to particular boxes on the wired lan, but thats not going to scale well… That was actually the main reason I wanted to use PF sense, I can get the owners to request fixed ISP wan IP address for now, and later if/when needed, they can request blocks of fixed IPs and I could route them internally to various offices using pf senses virtual IP functionality. The subject of vpns is bound to come up also, people will want to connect to AWS and GCS etc. I’ve done this in pfsense before so was trying to stick with stuff I am familiar with…
).
following some on-site investigation, I discovered each of the incoming 5 WANs are already coming in as tagged VLANs, and they are all set to 10 by the ISP. It doesn’t matter right now as there are 5 separate routers (ISP supplied). I am not sure the proposed plan would work now, as on the potentially new aggregator switch I will have 5 incoming connection tagged as VLAN 10. I don’t think the switch can somehow remap the VLAN tags on the copper ports so that I would wind up with VLAN 10/20/30/40/50 coming out of the sfp+ uplink port…