Hey all, I’m starting to do some upgrades to my little lab I got going on. My goal is to have my gaming PC/Workstation connected to my unRAID server through a 10G SFP+ connection using a MikroTik CRS305-1G-4S+. I want to be able to still access the internet/rest of the network but I want to make sure that the connection to the server is 10G speeds.
If the traffic has to go, let’s say (PC>MikroTik>UniFi>pfSense>UniFi>MikroTik>Server), would it slow to 1000BASE-T speeds once it hits the Unifi Switch and stay at that speed?
VLAN’s. I know there’s an error in there somewhere; go easy on me I’m still learning.
VLAN’s. Does this even make sense to do it this way? Am I over thinking it?
Well, you certainly don’t need a different vlan just for the sake of having it. VLANs make sense when you want to separate hosts from each other (e.g. because of security reasons). They are a neat way of grouping hosts logically into separate networks without having to run more physical cables.
And certainly, when the Mikrotik switch passes traffic to the Unifi switch, it can only do so at the speed of 1 Gbit/s and the connection (in the sense of end-to-end communication) will be limited to that lower speed. Therefore, even though the PC and the server are both connected via 10 Gbit/s to the Mikrotik switch, if they are on separate networks, the switch(es) will always pass traffic coming from either one of the machines to the firewall, thus reducing the speed to 1 Gbit/s.
That being said, if you had a switch with layer 3 routing capabilities, you could avoid the traffic leaving the switch even though the PC and the server are in different networks. The switch will then do the routing between the networks itself, therefore the traffic doesn’t leave the 10 Gbit/s segment at all. I believe your switch might be able to do that if you use RouterOS instead of SwitchOS.
You want the Unraid system on the same network as your computer or there will be a slow down as it has to go through pfsense. Short answer is don’t route storage, longer answer is in this video:
I have run into a problem. Seeing a while back that CentOS was pushing forward to 8.x, I decided to upgrade a couple of my servers to the now current V8.2. Both systems have 10GB LC fiber NICs (Qlogic using Broadcom 57810 chipset). These Qlogic card connect to a Microtik 10 port 10GB switch.
This all functioned while the servers with this fiber backbone were on CentOS 6.x or 7.x. Once the OS was upgraded to CentOS 8.x, the 10GB ports get marked as unavailable and will not activate. After being frustrated for a while, I checked on the Microtik to see if I had carrier/link. I noticed a couple of ports that had a fiber cable installed did not have a link light at the Microtik.
At the present time, I don’t have any VLAN’s defined.
172.20.20.1/24 is general internet/Lan traffic (Majority of systems and VM’s live here)
172.20.12.1/24 is the storage network (Pfsense to Microtik to servers)
172.20.30.1/24 is the work network (so I can isolate it in the Pfsense box someday)
192.168.1.1/24 is the WAN network (NG series ATT box to PFsense box plus a couple of Wireless)
Given the above information and the note that the 10GB seemed to stop working on the upgrade to CentOS 8.2 is there any chance that someo or all of this is due to mis-configuration in either the Microtik or Pfsense configuration? Like many before me, I found the Microtik configuration to be difficult at best. Yes, I found the driver for the Qlogic card and installed it. The card and both ports are seen in basic OS commands, like lspci and lshw. NMCLI knows they are there, but marks the ports as unavailable. I haven’t found a clear way to make them come to available state.
So, I never could get a connection the way the cards came. So I tried two things. First, in the cards BIOS, there seems to be 2 major mode settings. One I think was called SH and it seems to be a single connection with all of the bandwidth. The second was called partitioned mode. When I set it to this, I now have 4 connections per interface, each with two MAC addresses, a primary with the full card expected MAC and the second is the same MAC with the first two bytes set to 00 00. Suddenly, some of the interfaces sort of worked - well enough to pull an IP from the PFsense box. They still show as no carrier and Network Manager will not allow them to become active. So I ordered some DAC cables and SFP+ from a different manufacturer, thinking there could be some issue with either the fiber or the SFP+ modules. Swapping the SFP+ made no difference. The DAC cables did make a difference! Still have no carrier, but now the interface came active in Network Manager. Despite the interface being active and having pulled an IP from the DHCP server on the PFsense box, I still cannot ping either the gateway address or another interface address. Guessing this has to do with the interface not getting the full data, gateway address, route information, etc.
The card in all 3 servers are HEWLETT PACKARD 652503-B21 530SFP+ dual port.
Should this be set in the Partitioned mode? Or the other one?
I suspect the other mode is designed for iSCSI as I see iSCSI references in the log. iSCSI itself is not set up or running as best I know.
Comments on card mode and Network Manager set up would be appreciated.
Even though it pre-dated my first post, I attempted what Tom said about not having storage on a different network. That took me back to fiber/DAC ports being unavailable. Confused that when working, the interfaces could pull and IP from the Storage network DHCP server, yet appeared to have no route information and still would not work. I tried setting a manual route, but apparently did not point it at the right place…still did not work. I guess I need to document each step, so I can keep up with what has been tried. Annoying to see others having none to little issues and I ended up fighting the grizzly bear with a BIC lighter.
