My networking brain is failing me

Networking & Firewalls
1

Hey guys,

So I’m ok with networking, but I know I’m forgetting something obvious here, and its simply eluding me.

I cant get traffic to flow through one of the two “pathways” available. It works via the QSFP+/Eth0 interfaces on the hosts, but fails to ping when trying to use Eth6/Eth1 on the Synology/Hosts respectively.

My XCP-NG hosts can only see my storage server over path #1, and not both, nor #2.

Here is how things are physically setup atm:

image
image1291×589 99 KB

Each host has identical networking, so once I get it working on one, it should work on all.

To clarify:

Hosts:
Each XCP-NG Host has 5 total physical interfaces (0-4), 0/1 are 10Gig SFP+ (bonded, static IP 10.10.20.4x vlan 20), 2x 1Gig (not connected atm), and 1x Mellanox ConnectX-3 which is the management interface (10.10.55.x vlan 555).

Switches:
One switch is the CRS317 16port SFP+ switch, and the other is the CRS326 24port SFP+ with 2x QSFP+ interfaces (both are running SwOS, NOT routerOS). I simply dont need any routing features, only vlan tagging (and hopefully eventually MLAG when they release an update for SwOS with it).

The CRS317 is setup as:

image
image1054×549 43.4 KB

Ports 1-6 are attached to the Mellanox cards (Eth4) and are set to the management vlan (555),
Ports 8-14 are attached to Eth1 on each host (1 of 2 bonded 10Gigs),
Port 15,16 are for another switch (not in scope of this convo), and to my firewall.

On the CRS326:

image
image1321×980 83.2 KB

Ports 1-6 are again connected to Eth0 on each host again to vlan 20, and the the QSFP+ 1-4 (labeled Syno-7-1-4 to correspond to the Eth7 in Synology’s network interface).
Currently no other ports are used on this switch.

Storage:
My Synology nas has two native 10Gig interfaces, and has a compatible dual QSFP+ card installed. I have a QSFP DAC connecting interface Synology to QSFP+ Port 1 on the CRS326. My Synology has both interfaces bonded with (currently) 802.11ad LACP Balance-TCP. Which, based on this doc (here) This should work, without requiring any special switch config.
image

On the synology it shows as both interfaces connected.

I’m stumped as to why I can only get traffic to work over one of the pathways. Any help is appreciated here.

2

You can’t split bonded links using LACP across switches unless they support it. I’m guessing one of the links across the switches works, but the other doesn’t and this is why traffic only works across the one pathway or switch.

Setup both connections to one switch and see what happens.

3

Interestingly enough, Mikrotik just released RouterOS 7.1 stable, which include MLAG. I’m going to test it and see if that helps.

4

As @FredFerrel said above, but also, you are trying to MLAG 2 different L3 switch model here which is not supported (usually - I don’t know about Microtik) by the hardware itself.
To do what you want to do, you usually need the same model of switch, and you need to configure them a a “multi-chassis” switch. That will make them behave and seen as ONE switch to whatever equiment you are connecting to them. That also mean special redundant wiring between the two switch over fiber/10GE or above copper.

1 Like
5

Another option might be to see if Synology and XCP-ng support iSCSI multipath/MPIO.

6

Hey guys,I actually spoke with a Mikrotik employee, and they didnt recommend 7.1 for production. So for now I moved my entire storage to a single switch, with each host doing lacp on the same switch for now. I’m using RouterOS in bridge mode, because I was running into a flow control issues in SwOS mode. In RouterOS mode, it seems to avoid it by using the hardware acceleration properly.

I’m going to wait a few months until 7.x is in LTS, and then I’ll investigate buying a second one, and setting up mlag using the QSFP interfaces. until then, single switch it is.

1 Like