What Device/Appliance Should I Be Looking For?

Computer Hardware & Server Infrastructure Builds
1

I am visualising a device/appliance in my head that effectively replaces the DrayTech Vigor 2860 router/firewall/switch-combo device we have in the office whilst giving me the ability to install my own software. Requirements:

  • 1U rack mounted
  • 8-port PoE Gigabit Ethernet
  • Proxmox hypervisor:
    – Netgate pfSense for firewall/routing etc
    – Ubiquity Unifi Network Controller
    – PaperCut Print Services Controller

Does such a device/appliance exist? Is there a industry name for such devices? Or should I be looking more for a more traditional 1U server with the ability to internally add 2 x 4-Port PCIe PoE NICs?

If a 1U server, what would you recommend as a minimum spec to run the above within say 80% resource utilisation? I’m thinking maybe:

  • Minimum Intel Core i3
  • Minimum 8GB RAM
  • Minimum 256GB SSD, preferably M.2 NVMe

We have the ground floor of a small office building, EoFTTC Internet access, 20 or so staff all connecting by Ubiquity WiFi, couple of MFD printers.

1 Like
2

Supermicro has a lot of 1U models, but I don’t know of any that would also have a POE switch.

3

Thanks, Tom, nice kit, but even their cheapest products are over-specified for what I need, in other words, they’re too good! :wink: Interestingly, they don’t sell directly to the UK. What’s up with the UK?!

Looks like you can buy 4-Port PoE PCIe NICs - eye-wateringly expensive, though! 4 Port Gigabit PoE PCIe Network Card - Network Adapter Cards | Networking IO Products | StarTech.com United Kingdom

An company called Lanner appears to sell such devices, they call them ‘Network Appliances’, lots of examples here: x86 Rackmount Network Appliances. They probably cost more than the UK’s GDP but at least they exist.

Replacing a £200 device with a £1,000+ device is not going pass muster.

4

200 quid is your budget, then buy a consumer router basically what you have already. Since covid I’ve seen a good 50% markup on kit.

5

It’s not strictly my budget, it’s not actually even my money! But it is my recommendation.

I have a £350 ($430) NUC at home that does almost everything above (4-Port, not 8). I figured a 1U form factor with an internal power supply might push me out to £500 - £600 ($610 - $730).

6

Anything in the Mikrotik line up? What about the FS line up?

7

Mikrotik … interesting products! Never seen a manufacturer sell the boards inside their own products before! Broard line-up too. But, unfortunately, you can’t install your own software on them and they’re all ARM-based, which is a headache I just don’t need right now.

What is “FS”?

8

FS.com , they make switches and routers.

9

I would not virtualize PFSense, in an office setup

If you have issues with the host, or issues with PFSense you lose your internet connection which will affect endusers , especially now most things are in the cloud

No internet, unable to browse the internet for fault finding , download software etc

10

It’s a valid point and I wouldn’t ordinarily do this, I’m just faced with a particular set of circumstances that a virtualised ‘box of goodies’ would solve. I need to add:

  • A decent firewall
  • A Ubiquity Unifi Network Controller
  • A PaperCut Print Service Controller

All these product manufacturers want me to buy their expensive standalone devices and, ignoring the fact that they will not all physically fit into our mini-cabinet, they all have free versions of their software available for self-hosting.

We currently have an old, now EOL (no security updates), consumer-grade DrayTech Vigor 2860 acting as our single-point-of-failure Internet access; would a virtualised pfSense single-point-of-failure be any worse? That’s my thinking, but I do completely understand and agree with your point.

I’ve been running virtualised pfSense at home for the past year or so and, aside from an issue where pfSense would hang-on-start when proxmox does its weekly backup (now resolved), it’s been otherwise pretty stable.

11

What about a refurbed dell poweredge server? You can get them for pretty cheap and the hardware is rock solid. I have had my pfSense virtualized on my t430 for 5 years. The R2XX series is low cost and would fit in a small network rack, we have a few of them out there running simple VMs like this.

12

HP DL 360 (p or e) Gen 8 or newer also make good hosts, they are pretty quiet (for a server) when the loads are light (roar when at full cooling blast). The Gen 8 and maybe Gen 9 are tricky, there is some JAVA licensing garbage that gets in the way of some updates (behind a paywall), but mostly I’ve been able to find everything if I look hard enough (google search may not be your friend here :wink: ), use a different search engine for some things, but a lot shows up in reddit links.

The last 360p I bought were $200usd shipped and had 20 cores (40 threads) and 128gb of DDR3 ECC ram, you can run a lot of VMs off that much computer. These are Xeon E5 v2 so they should be supported in various hypervisors for a while. I no longer suggest going older than these processors, I was getting some odd things happening after a Windows update or two with the older Xeon X56xx series, this cleared up when I replaced the servers. With Linux/Unix VMs, it probably doesn’t matter.

13

We have a couple clients that buy these used/refurbed. Cost is so low they often buy 2 or 3 of them and have 2 in HA and one as a “cold spare”.

14

A very interesting development, something that had not even crossed my mind …

15

I have 3 of them (360p) in an XCP-NG pool for my lab, and also have a 360e with 8 drives for the Truenas shares. The storage is a bit slow, I think I need a newer controller because I should have about double what I’m getting. Just using a HP 420 in IT mode (JBOD) and it was tough finding the firmware and control application to get it into this mode. Eventually I should get something newer that will handle real SATA3 speeds.

Everything has 10gbps on short fiber to a Mikrotik CRS309 something something something switch/router, running routerOS right now. I put the same switch as a “top of rack” for my production system, this has DAC cables for everything and nice new Supermicro servers (Xeon Silver 10c/20t and 2x64 gb RAM), room to grow. Small handful of VMs is all I needed.

The HPs for the lab was simply a cost thing, can’t buy a decent mini computer with enough power to offer up a real simulation of a production environment for the same price, and I still would have wanted 3 hypervisors and a storage device. Trust me, I looked and looked and weighed the benefit of smaller, quieter servers for a long time before upgrading from the Dell C1100 I had before to the HP 360. I just couldn’t balance cost against desired performance. For my lab, these HP are way overkill, but they offered enough to make sure my production system would work at full demand before I bought the new servers for production.

And in a pinch, I could repurpose one of the HP to be a Guacamole server and allow outside access to around 35 of my student workstations for video/audio editing. Would take me about 1-2 days to get that running again if the world went sideways (again). I could probably even run our Tricaster through it and bring in ZOOM/Teams video streams and still produce assignments.

16

I assume you are in the UK. I don’t recommend trying to do this all in 1U we did this previously with the firewall and hypervisor. Issue is if there is server outage it takes out everything.

If you are on a budget get sophos devices and load them with pfsenee but I recommend pushing netgate as it comes with +. We did this for some cheaper clients and pair it with drattek modem

17

Here is my take on all this.

Part I - OPNsense revisited
I’ve decided to retry OPNsense that I had black flagged in 2019 for being really boggus and behaving often eratically while I was teaching network security in college. I was already a pfsense user at that time and was satisfied with it so OPNsense was new and tried it.

So last weekend, I removed pfsense+ that was running on a 8700k rig with 32GB and 2 x 10GE NIC that was doing marvels for my home lab, and installed OPNsense. After 30 minutes of fiddling with it and having discovered that I had to enable some tunable to bring up my Chelsio 10GE NIC, I also read that this problem existed since a few years already with OPNsense. I thought to myseld “this doesn’t bode well” if this is still maybe not properly recognized by the kernel. Strike one.

I continue setuping my network and when I was about to clean up some auto-generated rules, that’s when I remembered that this was an issue with OPNsense opening/allowing way to many things on every network. And you can’t just close those ports if you didnt need them. For me, every port facing the Internet must have zero opened port - so no IPsec NATing, etc. But there is no way to remove those on the WAN port. Same for the LAN ports who has other “built-in” rules. Strike two.

Time for some benchmarking as I always do when basic stuffs is in place. Well well, as in 2019 on slower links, even on the same hardware that pfsense+ was running, I couldn’t saturate the 10GE passing through it with a simple NATing rule. There is something in OPNsense that is wrong, or it could be the Chelsio driver that is really not optimized. Strike tree.

Please, watch out if you want to go OPNsense juste because you now hate Netgate.

Part II - Enters IPFire
Seeing that I will not use OPNsense with all the hickups it has, I’ve decided to retry IPFire that I was not really fond of because of the way to configure it and the lack of control while it runs.

So, after battling the setup again with the GREEN/RED zone thing - at least it recognized my Chelsio right off the bat… but of course : it is Linux under it - I am just configuring 2 networks (for now) just to test drive it again.

One thing that I really like about IPFire is the Pakfire and the tons of package you can use afterward from the command line. Very neat indeed. But, the ugly face of performance hit showed its face and I was kind of stumped. Between my workstation and IPFire, I was able to max at 9.41Gbps non-stop via iperf3. On the other side of IPFire, I was able to max at my ISP’s max bandwitdh of 3Gbps. But when I tried from my workstation to the ISP via a simple NATing rule, I couldn’t even reach 2Gbps on the same hardware pfsense+ was running.

I said “wait a minute!”, this is Linux how come there is such a performance hit with NAT with that hardware? So I started adding some more stuff to try to single out if it was the hardware or the software with IP blocklists (another neat thing built-in of IPFire) and there, I took another 300-400 Mbps hit on the performance! On pfsense+ with a FireHol L1 IP blocklist, there is not a single bit that drops from the speed. With IPFire, well, you saw what I just wrote.

I know IPFire is using IP Table, but I can’t believe the hit on performance. If you guys encountered this, please enlight me!

And with that, my adventure with other free Open Source firewalls ended at 4AM after I reinstalled pfsense+. I decide to go back and see what the fuss about that licensing, but I was able to upgrade back to pfsense+. When I went to register again (I wanted to see what would happened), this is was I saw:

Screenshot from 2023-11-02 22-09-43
Screenshot from 2023-11-02 22-09-431169×140 10.6 KB

Yes I already bougth a SG-3100 in the past, and I did register that 8700k rig when pfsense+ upgrade was available, so I don’t know.

Will I try other solutions? Well, I lied to you: I have a second network at home behind a Fortinet infrastructure and it runs very well - I am a Fortigate specialist btw - and it is quite easier to manage than pfsense for a price though.

Why am I wasting my time with pfsense then? Because I like it. It is the Swiss Army Knife of firewalls that just work and where I have full control.

Does Netgate makes bad decision? Yes like many other companies. But you know what: buy their hardware - at 189$ you have a decent home router/firewall that will last you 5+ years. And no more licensing drama either.

Do I have hope a new kind of license will come out from Netgate? Yes I do. If Netgate roams any forums here, or theirs own or Reddit, they will adjust.

TL;DR
Stay with pfsense, you wont regret it. Other solutions like OPNsense are still subpar. Buy a Netgate appliance as cheap at 189$ and be done with the licensing FUD.

1 Like
18

Kinda agree with that, ultimately they have hooked me, if I must pay then I will buy one of their cheaper hardware options. I’ve bricked a couple of routers in my time and donated away more than I can remember so you end up paying one way or another.

19

I have taken your advice, chaps, I’ve uncoupled the Internet connection from my virtualised services, you’re right, too big of an impact if it all goes down. I’ve plummed for an ISP managed rack-mount Fortigate 60F with UTM for the Internet connection, and I’m going to go for a 2nd-hand server for the virtualised services. I can deal with printing and AP configuration going down, can’t deal with a loss of Internet access, not these days, with everything in the cloud!

They said they’ll take on both WAN and LAN (including WiFi) configuration i.e. we tell them what we need, they make it happen, for £30 ($37) per month (3 year contract). This includes the UTM (only) license and the Fortigate 60F comes free. I can’t really grumble at that. This is part of a bandwidth upgrade to symmetrical 1Gbs, but we were going to do that anyway.

I would have liked to have had ‘my own little play area’, but I’m so busy anyway, this is probably the best solution.

Thanks for all your advice and stories - really appreciate it, chaps.

20

From my point of experiance and your requirements i preffere the following Scenario

  • TrueNAS Host

  • I’ve the Unifi Controller manuall installed in an FreeeBSD Jail on the Host with and NGINX reverse Proxy to connect trough HTTPS port 443.

  • Paper Cut can be also installed on trueNAS VM based.

  • Firewall appliance starting point can be an Barebone, fanless and externaol Power Supply without PoE.

In my case, i’ve the following specs for my LAN to DMZ Firewall in place.

  • Intel(R) Core™ i3-4005U CPU @ 1.70GHz
  • 4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads
  • 60 GB SSD internal
  • 4 GBit NICs
  • External power supply
  • Fanless device

The specs depence by your assumption based on office space (provider bandwith, RAM for Host and VMs)…

you can develop many ideas here. What I can already say is that you can’t reach the target successfully with 200 GBP for the listed services.

Regards
Andy