First and foremost, thank you both tremendously for your insights and taking the time out of your busy days to fill in this knowledge gap of mine. My apologies for the slow reply. $dayJob got in the way.
It is unclear to me how to reply to more than one individual in the To: line of a post. Since your posts are so tightly linked, I am going to @mention both of you in the the hope that you’ll both see this.
My project that we have been discussing this weekend went from “desireable future” to “decision needed immediately”.
My pfSense router at home, where the link aggregation is desired, suffered an irreparable hardware failure, requiring hardware replacement over the next few weeks. I am currently running pfSense on the only other 3+ NIC system that I have access to at present: a Dell PowerEdge 730xd with 256GB of ECC RAM. Not a tenable situation, since I need that machine for other purposes. I can live with it for a week or three.
What firewall/routers do you have in place now?
All my firewalls/routers are pfSense and have been since the project forked from M0n0wall. My choice of pfSense is more a function of my age and familiarity with the product than anything else. I have passing experience with deploying a few USGs at the houses of friends where cost was paramount. I am however open to replacing pfSense with another solution, learning curve pain notwithstanding, since I need reliability and speed more than anything.
At my house I was for historical reasons using one of those 4-NIC Pico PCs:
Intel(R) Atom™ CPU E3845 @ 1.91GHz, 4 CPUs: 1 package(s) x 4 core(s).
This is the router that just died that I am stopgapping with the R730xd.
Even if I do not change my current pfSense setup, I would want to buy the necessary server hardware that will in the future be able to handle the full load of, let’s call it an aggregate of 3Gbps, of external IPSec traffic to two sites: the bulk of the traffic will be routed via a physically nearby ISP coop, a smaller percentage to a somewhat farther away second ISP coop, which also will serve as the failover for the first ISP coop.
3Gbps sounds like a good target number with some margin: adding up all the links (assuming they all work), I am looking at downlink speeds of 2x 250Mbps (2x VDSL), 1x 270Mbps (Starlink on a really good day), and a whopping, highly variable and IPv6-only link with a peak 700Mpbs down via a 5G modem that recently replaced my old glacial LTE failover modem.
Call it an aggregate IPSec-encrypted downlink of around 2.5Gbps. So 3Gbps to be on the safe side.
That’s before any routing between my local VLANs. Virtually all of which not related to IoT devices are using 10Gbps links. The external IPSec requirements alone would place my home pfSense router into at least NETGATE 1537 territory. For that price, I believe that I an pick up a used 1U Dell, stick in a couple of SFP+ NICs, put a switch in front, install quiet fans, and get a real DRAC (which I happen to be a big fan of) included. Dimensioning suggestions for which PowerEdge to buy appreciated. I’d prefer something not more than 2 generations back. In the end I will need three physical routers, which I prefer to be identical hardware: one per site.
@brwainer I am not (yet) familiar with how one would use BGP to split apart one and the same TCP connection, spread it across links, and stitch it together at the other end. A pointer to introductory material on that aspect of BGP would be appreciated. I can consult with the networking experts at one of my coops for further advice, but I would like to ask this set of questions with at least some baseline knowledge in my brain. Do you have suggestions for some web pages or docs that I should read that cover this connection splitting/stitching back together aspect of BGP? Thanks!
Any and all advice you or others have to share at this point would be much appreciated, given that “need to purchase new router hardware now” has just become the #1 issue in my reality.