I know I may be asking a lot from the forum, but what the heck! I may as well try.
The MikroTik and any hard core routing is outside of my expertise, but I know enough to get hurt. What I am trying to do is offload the network / vlan routing duties from my pfsense router running on an old Dell R610 server to a new MikroTik CRS309-1G-8S+ running RouterOS. The setup of the MikroTik is completely foreign to me, so I really have no idea if what I am doing is correct.
Here’s a diagram of what I want, not exactly how I have it setup at the moment though.
ONe of the major questions I have is regarding the trunk from UniFi switch to the MikroTik across the bonded interfaces. In the UniFi port configuration (SFP1/2 LACP) I can pass All traffic, which includes tagged and untagged. On the MikroTik, I can only configure VLANs. Untagged traffic isn’t passed.
In the next version of RouterOS (v7, currently in beta) there will be some ability for hardware-accelerated routing (routing being done on the switch chip, aka a “L3 switch”): https://help.mikrotik.com/docs/display/ROS/CRS3xx+series+switches#heading-L3HardwareOffloading. This would do what you are looking for, but it is a very new setup that not a lot of people have experience with. I want to test it out but I don’t have any of the compatible models, nor really a use case for it right now.
Also no specific knowledge of the Mikrotik kit but;
untagged traffic over LACP link - Start tagging your main subnet with a vlan, it will be a bit of a pain in the arse to get all of the management vlans switched over but it will mean that all your traffic is tagged and your problem goes away.
vlan 50 being firewalled - Even if the switch does provide some form of firewall it will be so basic that you either might as well just pass vlan50 straight through to pfsense and have it route out or you will need to pass it through to pfsense to get the firewall flexibility to do what you need. I may be massively underestimating the CRS309 here I guess.
As a side question, is the Dell struggling to cope with the load or are you just doing it to pay?
Thanks for your thoughts everyone, it’s appreciated.
-=-= @LTS_Tom Thanks, I watched the video and now wish I had done that prior to my purchase. At least @brwainer gives me hope I can get HW accelerated routing going eventually.
-=
-=-= @brwainer - I made some terrible assumptions when I saw that it was dual OS, the 10G clouded my brain!! Figured it could route efficiently! I may as well play with the beta, this is just a home setup, so not super critical. (shh, don’t tell my wife I said that)
-=
-=-= @garethw Someone else (friend) had suggested that I tag all traffic, but as you said it’d be a pain. I’ll keep it in mind though. It would definitely resolve my issue.
In regards to VLAN 50, I suppose that I could pass that one VLAN to pfSense, although that puts me in the same predicament as with the LACP trunk from UniFi to MikroTik: tagged and untagged traffic. I’ll try and firewall that off using the MikroTik, there’s not a lot going on there so it should be able to handle the load.
The Dell is a beast and consumes a constant 110w, so I am working on reducing its load to the point I feel confident that pfSense can run on much smaller and much more efficient hardware. The Dell just needs to go. pfSense is a VM on a Windows Server 2016 Hyper-V host, so I also have doubts about it’s overall performance. I feel as though I’m not getting my full internet performance going through it. (Xfinity 1Gbps / 35Mbps).
-=
-=-= @mouseskowitz How so? Seems to be fine for me when I boot to it, traffic seems to be flowing around just fine. You may want to take whatever issue you have over to the MikroTik forums and support.
-=
Yes that’s the double edged sword of Mikrotik using the same OS, with all the software features enabled, on nearly every piece of equipment. Yes, that $20 router with a single ethernet port and the most basic CPU could be configured the same as the $1000 router, but it won’t actually perform doing those tasks. That’s why Mikrotik, unlike most companies, makes their block diagrams available for every piece of equipment. Looking at the block diagram to understand what the capabilities are is a vital skill for success with Mikrotik. There really isn’t a lot of handholding with it, because handholding requires more expensive development which isn’t Mikrotik’s style.
It is, sometimes, to get it setup but once its done it makes life easier.
Part of the issue is just being confident that you need to change the management vlan, then once the setting has applied make sure you are tagging the management vlan. Sometimes I have a couple of ports one untagged and one tagged and physically swap the cable, other times I change the tags, yet other times it just seems that I had the setup right and it was both untagged and tagged on the management vlan at the same time.
I got the crs326-24s+2q+rm instead of the Unifi 16 XG thinking I’d save some money and have more future expand ability. It’s been nothing but a nightmare. Yes it was easy to boot it up and get swap it over to SwitchOS. However, there is something with trying to use ESXi hosts on the switch that completely breaks it. VMs are not able to route traffic, and the really weird thing is that trying bricks the management interface of the switch.
The only way I was able to unbrick the management interface is do a netinstall to wipe the switch and start over. However, that didn’t actually work. So I had to console into the switch to make the reset actually happen. As soon as I reconfigure the switch and try using an ESXi host on it again, I’m right back to a bricked management interface.
I’ve tried working with Mikrotik support, their forum, and the seller of the switch. So far they say there’s no problem and I can’t return the switch. I’m waiting for the Unifi switch to come tomorrow before I mess with the Mikrotik any more since it’s at least moving packets to some other devices.
I’ve now spend 12+ hours of my time trying to save $100. If they don’t let me return the Mikrotik, I haven’t decided if I’m selling it or blowing it up to vent my frustration.
