The hypervisor, storage and backup servers has 2x sfp+ interfaces and are physically connected to both Mikrotik switches (1 port per switch). At the software level, we setup the 2 ports as a bonded interface using the LACP protocol and we setup the mikrotik port interfaces across both switches using MLAG.
I cannot help but think that something is wrong but I cannot figure out what yet…
Do I need to swap the wan/lan interface so i have 2x sfp+ in lag in firewall and then every firewall has 1 lan port in each swith?
Right now your only single point of failure is the core switch and whatever is upstream of it. For true redundancy, you would have two ISPs with diverse (not sharing a physical path as much as possible) circuits. This could be a fiber ISP and a Cable ISP, or it could be two DIA circuits that were ordered from the same company as a diverse pair. This is where things start getting expensive, of course, so you have to determine where to draw the line.
Separate from that, I would suggest that you connect each firewall to both switches with LACP/MLAG just like the servers have. This is more of a traffic optimization than a redundancy improvement. Right now the servers are being told they can go to either switch to reach the default gateway, but the active firewall is only present on one of them, requiring traffic that hits the wrong switch to traverse an extra hop.
You already have a direct connection between the two MTs to facilitate the MLAG, right? If not, that’s a mistake - I just assumed it was there but not drawn because it isn’t part of normal packet flow. MLAG, or Cisco’s VPC, or Dell’s VLT… lots of names for it… always needs an SFP+ or better connection between the two separate switch planes.
Having this connection, or any other extra connection between the two MTs, would not replace a full cross-connection between the left OPNSense and right MT and the opposite.
So If I understand what you are saying properly, I need to connect the 2 switches via a dac cable MT1 sfp+16 to MT2 sfp+16 keep opt2 interface for pfSync, move WAN to opt0 interface. bond ax0+ax1 in LACP, connect ax0 to left MT and ax1 connected to right MT.
Then create my LAN and Storage interface as 2 vlan on top of the newly created bond
Yep, sounds correct. When you make the connections between the two MTs, you should review the documentation on MLAG to make sure it is used for that and also that you don’t create a switch loop. I hope you’ve got some form of STP set up across the board.