Which head end in a nightmare project for my first client

Computer Hardware & Server Infrastructure Builds
1

Good morning all,

Full disclosure i’m posting this in a few forums to gather information, and so if you are a member of many forums you may see this thread pop up in duplicate.

I need help, not really afraid to say that … i need to provide a good solid solution and want to do due diligence and full research to dig in to pros and cons.

So straight into the Background

  • Small to medium business center
  • Multiple tenants
  • Multiple tenant spaces

The services are currently:

  • 3x VDSL lines each with 5 static IPs
  • 1x fiber line with 5 static IPs

Peplink Balance 380 balance/router/VPN feeds out a single LAN connection to 6x generic netgear switches, configured with VLANS from the Peplink in segments.

No warranty on all devices, experiencing serious slowness as the peplink only has the ability to do 140mbps throughput. Constant drop out of lines due to high CPU load. Switches are making some serious rattling noises, and no one has the password to them only the config layout in a spreadsheet of whats setup on the ports VLAN wise.

Current use case:
The building provides internet in the form of a VLAN for each tenant space with an associated DHCP scope. Some tenants use the wall sockets in their spaces to access the internet, some tenants install their own routers to a single outlet and configure their own networks. This is inherited over many years and is just the way it is.

A few tenants pay for a static IP address and attach VoIP to this as well as other options. The business center does not allow the installation of tenants own internet lines.

Proposed
Replace all infrastructure due to warranty, age and lack of access. WiFi is not currently in scope as this is not provided currently to any tenant, this is another reason tenants use their own off the shelf garbage routers in the building.

Replacing with a full unifi stack of switches, most likely an aggregation and 4x48 port variety.

My question is the head end. The top of the stack. What should i go with?

UDM Pro? i realize is not the worlds best, however it is what i know. It obviously doesn’t do multi WAN connection, just a single WAN and Fail-over. And so this would be the first change that all tenant’s would have to change their Static IPs to the main line in.

pfSense?, Would i be able to keep the current balancing etc with a pfSense box like a 6100?

Another solution?
Would keeping the Peplink balance purely for link balancing and aggregation, then feeding to a UDMP work? The UDMP allows multiple WAN addresses now and so this could potentially work? And all the VLAN could be done there.

Or another solution totally …

Please be constructive with comments etc, i’m only in the planning stages however the project will most definitely go ahead in one way or another. I would rather not push the business center and all tenants to completely change the way that they work. This i believe would not happen.

1 Like
2

I would go pfSense as it’s capable of the advanced configurations that will probably be needed for this all to work smoothly.

6 Likes
3

I cannot agree more with Tom. This seems like and advanced configuration, also for the fact that the requirements can change if the Business Centre’s customers change.

I would suggest you to stay away from the UDM and all of Unifi’s firewalls. We only put them in homes for non commercial use in case the ISP’s router sucks more than the USG/UDM.

You can go with pfSense if you have the skills to set it up and manage it, or you can go with something like a Sophos XGS 2100 if you want a lot of NGFW features and a Cloud manageable firewall. It depends if you want and need all the NGFW.

5 Likes
4

This sounds very much like a serviced office I used to look after, with the added complication (in my case) that the sales guy had bought all the kit then told me to “design the network based on what we bought”. Great, thanks.

Anyway I must have done an ok job as 8 years later the newly installed network was still working fine.

As Tom and cabassi said, go with pfSense to do the balancing and firewall duties, add the VLAN interfaces here and then the switch config is pretty simple with just trunk ports up to the aggregation switch and pfSense, and the rest access.

Feel free to DM if you want to talk through ideas.

1 Like
5

@pierschip wow thank you very much indeed for the offer. Something I will definitely take you up on if that’s okay. The design I don’t feel I should try to reinvent the wheel on really bit there could definitely be improvements made.

I think we have decided on either buying an SG6100 or using a 3ghz, 8gb ram, 256gb SSD dell micro desktop with 4 port gigabit intel PCI card install. I would prefer the netgate hardware but of course if there is money to be saved then I guess it’s not a terrible trade off.

6

I’ve dropped all the Peplink products that I had installed previously with VDSL WANs as soon as FO became available. Very few vendors seem to be making firewall hardware that can keep up with FO speeds and low latency. Agree with the Pfsense recommendation. I use Untangle with hardware running i5 processors or higher for business environments like that. Just my opinion.

1 Like
7

So my brand new SG6100 arrived yesterday for me to configure. I’m gonna try this myself and see how far I get, I have all the Netgate course and docs to look at, plus hopefully I’ll be able to get people on the forums to help with anything I can’t understand.

Guessing I can just simulate 3 different WANs by presenting 3 internal IP ranges to the ports and pretending.

I’ll keep everyone bugged with my questions :ok_hand::metal::+1:

2 Likes
8

Okay so just keeping this thread alive. @pierschip I’ve started and am already through a small part of the configuration dump from the old system. Documenting all the details for the few tenants that require more attention.

I’ve started my mastering pfsense courses and started wading through the documentation on the netgate site. Some serious assistance might be required when it comes to transposing this knowledge and setup over.

As a backup I’ve already got in mind that I would have to pay for some consulting time with Lawrence systems to help if it gets bad lol

2 Likes
9

Okay so we are on … I won the bid to get this project done. And what’s even more awesome is that I not only have the first contract for on going support for the next 3 years, but I also get to change the setup anyway I want to make it run better.

So I’d like to maybe throw out for discussion based on the scenario on this thread how people would do this differently or in a better more easily managed manner.

They have signed off the following

1x Netgate 6000 series (the new one)
5x 48 port unifi gen 2 switches
1x aggregation 10g unifi switch

I’ll be using an Intel nuc i7 16gb for the controller

And another nuc for greylog

Looking at controlling and balancing 20 static IP addresses over 4 WAN connections

One primary 500 Mbit fiber line and the other lesser vdsl 50mbit services

Come at me.

1 Like
10

That sounds like fun as much as it does profitable, very nice win!
The nuc for the controller is certainly overkill for 6 unifi devices, I would consider running it as a container so you could throw more services at the nuc as needed. Such as Apache Guacamole behind haproxy + authelia for 2fa access. Which by the way, pfsense has haproxy as a package ;-).

If you get fancy, then you can put both nucs into a cluster and have some failover. Which then it would be good to learn either kubernetes or docker swarm. But at least if you start out with a simple container, it gives you the option of growing very easily.

**You can user built in features of kubernetes or docker swarm for failover, or use haproxy as a load balancer depending on the service.

1 Like
11

Okay everyone, very much keeping the thread alive so that i can maybe lean on some more knowledgable people for the pfsense config. Even after following a few “Mastering pfSense” courses for the last few weeks i feel they were all woefully lacking in my specific requirements.

The config phase of this project is fast approaching, the hardware arrives this week (w/c 11th October 2021) and the config and setup begins w/c 25th October 2021.

I’ll be building the entire stack out in a separate rack next to the existing system i hope so this will make my life a lot easier, i plan to only cut over when im sure the config is as sound as possible.

Multiple WAN IP directing and load balancing are my main knowledge gap concerns here.

1 Like
12

Okay first question that maybe someone here has knowledge on or @LTS_Tom you may know of the top of your head.

The SG-6100 has 4 WAN ports on, 2 of which are duel copper or optical and then two other optical only. Can I put copper 1gig sfps in then all and have 4 incoming WANs as this is what I need

13

There are two 1G combo ports for either 1Gbase-T or SFP connections. Those you can’t use both.
But the other two 10G Ports can be used as WAN/LAN whatever you like. The pfSense is very flexible on that. So you could have 2x 1GB Copper as WAN and use the 10G Ports with a Copper SFP as WAN too. But I wouldnt recommend using T SFP’s better hook up a Switch via 10G SFP+ and put all your incoming WANs into transport VLANs.

14

GBC_New_NoIP_NoVLAN.drawio

How would you change this initial layout if i might ask, Transport VLANs are not my thing, nor is pfsense to a great extent.

15

Also (questions coming thick and fast here) i have the following scenario(s) i need clarification on

Tenant1 has a static IP from WAN2 using for VoIP
Tenant2 has a static IP from WAN3 using for Webserver with port forward
Tenant3 has a static IP from WAN4 using for various systems and ports

Load balancing with the old peplink for these tenants was not possible as system break when they are not ingress/egress on that IP they are expecting. On Peplink this was solved using persistent or as they call it Enforced rules for that IP. I assume hopefully that pfSense deals with this in much the same or similar way?

16

Day 1:
So day one was a bit of a right off in terms of progress. It took the delivery company all day to deliver one APC rack to the second floor and broke it in the process.

Tomorrow I’m hoping to get the rack in and the switch layout finalised, also cut over from the current all dirty power setup, to the duel APC 3000 UPS boxes instead. Then I can look at basic pfsense internet, controller setup, adoption and VLANs on the unifi side day 3.

17

you can define outbound FW or NAT rules to force special ports / IPs to special WAN Interfaces.

18

Day 3 : getting irritated.

Thought I had this figured out tbh, but doesn’t seem like my thoughts were well founded.

Got the VLANs setup on the unifi side, got my VLANs and DHCP servers setup on the pfsense side.

On the pfsense box I have the WAN working and the Multiple additional IPs for that line added under virtual IPs.

I have configured a port with just the VLAN I want on the unifi stack and it it all works through and out to the internet. I thought I had the Virtual IP and NAT configured correctly to pass all traffic bound for one specific WAN IP to the IP internally it should be going to. Nothing.

Am I missing a step? When I use the machines in that VLAN and browse to what’s my ip, I get the correct external IP address, but anything coming from the internet TO that IP address isn’t getting through to inside

Ummm … Help?

19

Hi All,

Going wrap this thread up, many thanks for eveyone involved in pointing and nudging me in the right direction.

I sit here on Day 1 of go live arounf 1pm in the afternoon, having spent a large portion of the morning fire fighting rogue setups etc. God this place has more routers than a datacenter it would appear.

pfSense is running well although not load balanacing anything as yet, people really arent noticing since the main line is 500M+ down and 50M up. Those with statics on the lesser VDSL lines are now seeing improved performance now i can direct traffic away from them.

A few small internal routing issues to be ironed out like traffic inside one of the vlans cant resolve to their internal webserver, but im sure i’ll be able to figure it out. Access to the website hosted from outside works (but slow) access to the website internally from behind their router seems to be not working … weird.

Anyway again thanks to everyone on this huge learning curve opportunity. Anyone starting their own business i dont suggest implementing a system you have no idea about … but hey it forced the learning curve like never before lol.

2 Likes