Hi Folks - this probably falls under “I forgot something simple” but here goes…
I have a Netgate 6100 and UniFi switches. “Santa” was good to me and I picked up a USW Aggregation. The primary reason was to move my storage network (VLAN172 - 172.16.1.0/24) which uses NFS between my Proxmox servers and Synology RS1221+ NAS (direct 10 GbE links right now - suboptimal as I want to set up a “proper” Proxmox cluster with shared storage) as the VMs are stored on the NAS. However, my ISP has now offered to upgrade my fibre from 500 Mbit/s to 3 Gbit/s and want to move my “core” network switch from being the USW 24 G2 1GbE uplink to the USW Agg 10GbE.
My plan was to move the uplink port on the Netgate from icg0 (1Gbe) to ix0 (10Gbe) and move the uplink (downlink?) port on the UniFi swith from the USW24 G2 to a port on the USW Agg.
Current configuration:
current “core” switch is a UniFi USW24 G2
port 1 on the USW G2 is the uplink to the Netgate
native subnet is 10.100.200.0/24 (a/k/a CoreLAN200)
Other VLANs (name includes VLAN number) Servers20 (192.168.20.0/24), TrustedWired25 (192.168.25.0/24), TrustedWiFi30 (192.168.30.0/24), etc.
USW24 G2 port 1 is configured with CoreLAN200 and tagged VLAN Management Allow All
Netgate has CoreLAN200 on igc0
Netgate has Servers20, TrustedWiFI30, etc. are also igc0
uplink between USW24 G2 and USW Agg is configured the same as between the USW24 G2 and the Netgate
proposed uplink between the USW Agg and Netgate configureed the same as between the USW24 G2 and Netgate
all connections are using 1GbE and 10GbE DAC cables as applicable
have STP RTSP priorities set as appropriate
After backing up the Netgate and UniFi configurations here is what I attempted:
using my “management” port (igc1 - 10.100.101.0/24) logged into the Netgate
moved CoreLAN200 on the Netgate from igc0 to ix0
changed the VLAN assignments from igc0 to ix0
saved the settings
unplugged the USG24 G2 from Netgate igc0
plugged USG Agg into Netgate ix0
Results:
network failed with connectivity loss
some UniFi switches defaulted to 192.168.1.20 address
rebooted the various switches and Netgate - issue persisted
had to revert to original configuration
I am not sure what I did wrong here. I could be something simple…
Thanks all!
Added - Images to better illustrate.
UniFi Current/Proposed Configuration - Red line current/Blue line proposed
Just to be clear, with only the USG-Agg plugged into the Netgate, you have no network connectivity?
By which I mean, the USG Agg is not connected to anything else (except maybe the device you’re testing with). No downstream switches, including the original USW24 G2.
I was thinking the same thing. Maybe it wasn’t made explicit, but either (a) the USW24 would need to be replaced by the AGG, i.e. the downstream links to the other switches would need to be attached to the AGG, or (b) the AGG would be for cascaded in front of the USW24, i.e. you would attach the USE24 to the AGG instead of the netgate.
No, I had an annual leave day and since my wife was at work and my son was at school I was free to experiment. Change management at home is harder than at work
The USW24 G2 is attached the the Agg using a 1GbE DAC. Everything else is a spoke of the USW24. Thus, it would be:
Netgate ↔ Agg ↔ USW24 ↔ rest of switches (DAC or fibre) ↔ rest of network
Currently it is:
Netgate ↔ USW24 (RJ45) ↔ rest of switches (including the Agg using DAC, rest DAC or fibre) ↔ rest of network
I was thinking about taking the pfSense xml backup file and manually editing the interfacing and then restoring the new configuration. Not sure if that will work. (My wife and son are home so testing that could cause agitated “clients.” )
I change the STP priority of the Agg switch to 0 and moved only VLAN20 (192.168.20.0/24) over to ix0 (no other VLANs or the “core” subnet 10.100.200.0/24) and it was not accessible from the other VLANs (25 or 30).
I move the IoT VLAN (VLAN 101, 192.168.101.0/24) over to ix0 on the 6100. In the UniFi controller, it shows connected at 10GbE but the 6100 is showing Status no carrier. See image below:
(1) your USW24 does not seem to connect to the AGG using 10G. This seems to be wrong. But that is not the big problem here.
(2) you could leave the AGG on the side for a moment and try to connect the USW24 with 10G to the netgate, just to be sure where the problem actually is. If this works you know it is somewhere with the AGG. If this does not work, it is maybe something with the SFP+ module you use with the netgate? I am just speculating
The USW24G2 does not have 10GbE The link light is on on the 6100.
The DAC cable that I am using is a 10Gtek SFP+ DAC Twinax Cable, Passive. The model is CAB-10GSFP-P.
I tried a second DAC, same model, and Wiitek SCP-H10GB-CU1M that was between my Snyology NAS and an HP DL380 G8 (known good). The link on the 6100 is on in both cases but in pfSense the link is still down (no carrier).
Just to make sure I am not doing something dumb (or mis-reading the Netgate docs) is this the correct port assignments?
here is what I attempted:
)
The link light is on on the 6100.
