Moving to Fiber Based Network Storage

Computer Hardware & Server Infrastructure Builds
1

Hello,

I hope this e-mail finds everyone well and everyone is able to manage all of this snow we are getting here. I think Buffalo, NY has over 4x feet in some areas after just 24-36 hours.

We have individual 1U rack mounted servers from Dell and Lenovo. The Dells are RAID1 and the Lenovo are RAID5. We are currently buying more Lenovo servers and we would like to make the move to fiber-based storage. This will make it easier when we migrate to XCP_ng from ESXi. That’s our goal.

I have been looking at 45Drives and Synology. We are not a big company, under 20 people, however we really like the idea of XCP_ng having the virtual servers hosted by a central storage solution. We would like to order the new Lenovo server so they come with a minimum storage configuration. Maybe installing 2x M.3 drives simply for XCP_ng and then linking the virtual servers to the hosts with fiber. Then we can start to migrate everything over from ESXi onto XCP_ng servers.

What I’m looking for is a reasonable way to do this, without spending a lot of money. We are not a very big company, so I’m hoping there is a low cost way to do this. What I’m looking for is for suggestions on:

  1. For the physical hardware Dell and Lenovo servers, what would be a good fiber storage card to purchase at a reasonable price?

  2. What would folks recommend for the fiber switch? We wouldn’t need a very big one - we will have approximately 10x physical servers connected + the network storage.

  3. And finally - what would you recommend as the best storage system to go with? I’m open to a number of things - including going TrueNAS if that makes sense, but to me, I think that means I have to buy another server and then install RAID5+ on that server. I’m just looking for something that is reasonably priced and will give us anywhere from 10TB to 30TB and multiple drives. I want the ability to not have to worry when a single drive goes bad - it’s just a matter of purchasing a new one. In this case, we normally purchase 3-6 extra drives at the time we purchase the server/storage device. I would like the ability to not have to worry about that in the future and just install any new drive the meets the same specs or better for the faulty drive. Any thoughts on that would be appreciated too!

I’m trying to just keep this simple and cost effective. We really like the idea of a central location for our virtual servers to be “stored” and move away from the physical servers storing the virtualization locally. The only other thing we would like is for the SAN/storage solution to have a SMB share or some type of folder where we can store ISOs our company uses and software backups.

I really appreciate everyone reading this and any suggestions that are made would be GREATLY APPRECIATED! THANK YOU SO MUCH!

Thanks again,
Neil

2

First, not clear on why you want fiber, generally people ask about speed. A common setup would be SFP+ which are 10G and support both fiber and DAC connectors. Next speed up from that are SFP28 which supports 25G and also works with DAC or fiber.

Dell and Lenovo both make good servers, XCP-ng and TrueNAS work great. We use a lot of 45Drives systems running TrueNAS for storage. Because they are easy to manage UniFi switches are good, MikroTik are good budget switches but come with a much steeper learning curve.

I have a video explaingin DAC and fiber here

I have a video explaining storage design here

I have a video showing how to setup XCP-ng NFS and TrueNAS

For a general overview of how virtualization storage works in XCP-ng

1 Like
3

Fiber Channel or Fiber Optic, just want to make sure we are talking about the correct thing.

I haven’t done much with fiber channel and SAN devices, so not going to speak on it.

I do run my small XCP-NG system on a NAS over both SMB and NFS which seems to work fine for me. It might come down to how many databases you are running, and whether they use lots of small writes/reads or large writes/reads. You may also want to test iSCSI for your use case if you go this way. I’m currently running everything on 10gbps ethernet, but you could go 25gbps or faster if you think you need it.

1 Like
4

Thank you for all of your help and assistance. You have really helped me along with the missing pieces in my plan.

Just a question… I’m still planning on going fibre. Just curios if there is a fibre card (I’m assuming PCI for the servers) you would recommend for the servers and/or the TrueNAS box I plan to buy/setup? Lastly - is there a low cost fibre switch that you like or would recommend??? I’m not necessarily trying to do this at the lowest price, but balance up time with as low of a cost as I can make it (safely).

Again - I think you for all of your help and support - it means more to me then you will ever know - THANK YOU AGAIN!

Neil

5

For 10G DAC cards the Intel X520-DA2 are good and well supported.

1 Like
6

Fiber Channel or fiber optic? The answers to your above questions depend on that answer.

For 10gbps ethernet (with or without fiber optics), there are a few Mikrotik switches that I like. For small 10gbe the CRS309-1-8s+in works well, I just pulled mine out of service to expand. Expanded to CRS326-24S+2Q+RM. I come out of this to an old Cisco 2960s with SFP+ (DAC cable) because the Cisco has POE+ and plenty of rj45 gigabit for those needs.

For cards, I’ve recently been buying used SuperMicro AOC-STGN-I2S Dual Port 10GB SFP+ 10GB Card Rev 2.1 Low Profile for the computers that need it, if you buy rack mount servers, going with the manufacturers 10gbe solution can be best. My big lab has HP servers and HP 10gbe SFP+ boards installed. These boards are specific to these servers.

I’m using a lot of used Cisco SFP+ multimode SR 850nm optical modules and short OM-4 jumpers with some 10Gtek DAC cables for other things. I also just put a QSFP to 4xSFP+ cable on the new switch going to the mini lab.

I do have a single device on 10gbps copper and bought a 10Gtek SFP+ 10g-base-t module, these things run HOT! I do not recommend running your whole system on 10g-base-t due to power draw and heat. DAC and multimode or single mode modules run a lot cooler.

All of the above is regular common networking, not Fiber Channel and I don’t think the switches listed have the option to switch Fiber Channel even if your cards can do it. Maybe FCoE (Fiber Channel over Ethernet) but I have no experience with that.

Almost everything in my lab is bought used on ebay, I think the DAC cables and OM-4 jumpers are the only parts that were new besides some marked down auction surplus HP T740 that I got really cheap, go with T755 if money and supply allows (two extra cores or 4 extra threads in the T755).

DIY Truenas or Truenas Mini is really the only choice for a Truenas system, all the bigger “enterprise” level of stuff they sell is going to be a lot. If this is just a lab, DIY is probably the way to go. If you look at HP servers, I only suggest gen9 or newer, the older generations do not support UEFI. Also be aware that gen8 and gen9 can be difficult to find drivers and BIOS, for a while HP had this locked behind a support contract. I think all they now need is a free account. My guess is that this has to do with the JAVA licensing changes and their iLO function, just a guess.

7

Hello everyone,

I hope this message finds everyone well!

I accidentally posted this topology on another ticket I have open. It didn’t make sense to put this topology on that ticket/matter/question. Anyways, I’ll re-post it here.

This is the current design I have. I don’t know if it’s good or bad, right or wrong?

This is how I think the network should look when I’m finished. I’m uncertain on having two separate switches, one for just regular IP network routing and the other switch for the fibre connectors only and storage - basically a dedicated switch to bring all of the servers PCI fibre boards into a central location where the TrueNAS or 45Drives unit will also be connected via fibre.

One of the alternatives to this is to use my existing chassis switch and add a fibre card and dedicate all fibre ports to a dedicated VLAN that has nothing to do with any of the regular data networking, routing, security, wireless, etc.

My design over all would be to have M.3 drives (2x, RAID1) in each server. XCP.NG would be installed on there and all of the virtual servers running on that particular host would be stored/saved on the TrueNAS/45Drives box. I don’t know if I should go with 45Drives? I really like the idea of supporting a fellow Canadian company.

I’m eager to hear everyone’s thoughts on this. I hope I’m going about this the right away.

Anyways - I really do GREATLY APPRECIATE everyone’s help on this.

Thanks again,
Neil

New Network Topology
New Network Topology1028×782 220 KB

8

I think this plan looks sound.
I also use a dedicated SFP+ switch for storage, but I think if it is less expensive to add SFP+ ports to the existing switch it shoudl be fine too just going with one switch.

As SFP+ cards I’d get used Mellanox CX4121A ConnectX4-LX 10G/25G

You may want to use 2 SFP+ ports on all servers (in addition to 1 port CAT 1Gbps for management)

  • hypervisors: 1 port for storage, 1 port for warm VM migration (vMotion)
  • storage: 2 or more ports for storage, because you have 8 hypervisors
1 Like
9

Thanks for mentioning those cards specifically. I look them up on Amazon and they come in around $79 - which seems pretty good. When I look at it on Amazon, I’m unclear if it comes with the GBIX or not???

URL to Amazon:

Also if I’m not mistaken I can either order the DAC cables or true fiber if I wanted to go that way. Also, I believe the DAC cables will do up to 10GBps via copper and fiber can obviously go faster than that, but I don’t think at this point there is a need for faster fiber other than 10GBps fiber…??? Any thoughts on hat???

I think the only thing I have to figure out all of this, is what I’m going to use for storage. Should I go with TrueNAS or should I get something from 45Drives. I understand 45Drives has a Home-Based Server solution - so I’m not sure if maybe that’s a good place to start for storage? Maybe I should give them a ping. I’m just trying to keep costs as low as possible, while going with fiber so that everything is using the same technology gone for (i.e. when I add more servers).

I’m eager to hear from anyone on this solution… Any suggestions is very much welcomed - THANK YOU!

Take care - and I hope everyone has a safe week!
Neil

10

The port type and adapter determines the speed, not the medium. The SFP+ is 10G and SFP28 is 25G and they make DAC and Fiber that can support both speeds.

The 45Drive Home Lab Series is great and they run TrueNAS well.

11

I’ve been buying these (from this seller) SuperMicro AOC-STGN-I2S Dual Port 10GB SFP+ 10GB Card Rev 2.1 Low Profile | eBay

Same chipset as x520 and generally speaking “unlocked” unlike some Dell, HP, etc.

Not sure if you can find full height brackets, everything I’ve needed recently has needed half height. I bought 5 of these from this seller, so far, so good.

12

Just to throw a cheaper option out there, I would look into building a single server large enough to run all your VMs. You can back everything up offsite to AWS or Azure and also have another box onsite where you can replicate to each night if you need faster RTO.

You already have single points of failure with the SAN/NAS and a single switch so you wouldn’t be any worse off and you’ll also have less complexity.

As an example, I can build a Dell R440 with 2x 3.0 GHz 6154 18-core procs, 256 GB memory, 8x 1.92TB NVMe drives (I would run in RAID10 and use for datastore) and two M.2 SSDs (use for XCP-ng install) for less than $4100. If you need hardware support Dell might cover it, but there are plenty other 3rd party that can provide next day parts if need. Also, NVMe drives have a 5 yr warranty.

13

Thank you @xerxes for your input - I GREATLY APPRECIATE IT - THANK YOU!

I’m a little confused with what you said in terms of the last two bullet points. I’m thinking I would use either 1x NIC or 2x NICs trunked together with all of my VLANs assigned. This is where I would connect to the VMs, manage the VMs and the VMs would use this connection as their primary “virtual NIC” that goes to the firewall (Internet) or to another PC/server on the LAN.

From there, there would be a single fibre cable running from the fibre card on the server to a fibre port on the fibre switch. I thought this would be enough for accessing the storage off of the NAS. There was a consideration of having two fibre connections from the NAS to the fibre switch.

What do you think of what I said above? I’m hoping I’m explaining it clearly enough.

Thanks again,
Neil

14

Hey @FredFerrell - it’s something to consider. Right now we have over 60x virtual servers and that number is going to only grow. I can’t see us moving to a single solution - we already have6x Dell and Lenovo servers all running VMware. We just need an additional server to take some of the heat off of some of the other servers and give us a little bit more room to expand additional virtual servers. I don’t expect it will be too long and I will need another Lenovo server.

Thanks for your help - I do really appreciate it!

15

Hey @Greg_E

You mentioned something that caught my eye. You talked about the fibre cards running hot. Is that an issue you see a lot of? Our data centre has 365x A/C from the roof. It’s built to provide AC even in the winter (although that seems strange in a way). That’s what we have. Just wondering if the fibre cards are going to generate a lot more cost and heat that I wasn’t aware of? We do have other systems running fibre, but that is for telephony and not for IP. And we have the fibre connections in from our IP carriers. I haven’t noticed them hot, but just wondering if that is something I should be worried about???

Thanks Neil

Just to bring clarity - I’m thinking of running 1G CAT6 connections from the servers to the chassis switch for their regular IP traffic. The fibre is just for the storage and nothing more. Completely separate network.

16

The 10gbase-t modules run hot, try to avoid cat cables for 10gbps if possible, stick to fiber optic or DAC.

2 Likes
17

Do refer with this to each hypervisor or to the NAS? They have different requirements.

Let me explain:

A storage system has many client machines, and in your case 8 hypervisors that simultaneously and permanently access shared storage on the storage server. In that case it makes sense that the storage server has 2 or more 10G network interfaces bonded / LAPC so that the client machines can use the bandwidth better. Only the storage server needs to have the bonded interfaces. Now we come to the matter of networks used on the bonded interfaces. You want the storage traffic to NOT be routed for performance reasons. You would have one dedicated VLAN just for the storage traffic for the hypervisors. This VLAN would go on the 10G bonded interfaces which are connected to a 10G switch. Also you need to manage the storage server, so you need a VLAN for management. This does not need to have 10G speed. Normally server boards already have some on-board 1G interfaces and you could use one of them for the management VLAN. The management interface would be connected to the same 10G or to another switch that only has less than 10G. This way you just have the storage VLAN on the 10G interfaces for maximum bandwidth usage. If the storage server has no onboard 1G NICs you can just put the management VLAN on the 10G NICs together with the storage VLAN.

Each hypervisor is a client machine to the storage server. If this is a server board you have surplus 1G interfaces and can use one for the management VLAN. If this is not a server board, jsut put the management VLAN on the 10G trunk interface. The 10G trunk interface carries the traffic for all the VMs and it shoudl be sufficient if you use 1 10G interface just for that. If you have two 10G interfaces you wan to reserve the second NIC for warm VM migration between hypervisors and have a dedicated migratoin VLAN. VM migration is quite slow if you don’t have a lot of bandwidth for this and you don’t want the normal VM traffic to interfere with the migration traffic. Bottom line is that you have one optional 1G NIC for the management VLAN, one 10G NIC as trunk for all VLANs used by the VMs on the hypervisor and an additional 10G NIC for the dedicated migration VLAN.

1 Like
18

I was thinking you had less servers since your company size was small. There are some good hyperconverged solutions out there like VMware VSAN or Redhat RHHI which would scale better.

I guess my only concern would be if you would have redundant controllers for the storage and redundant switches. If you do, then a centralized storage architecture would be fine.

1 Like
19

Hello everyone!

Thanks for the feedback from everyone that has posted here - it is very much appreciated!

Just a quick question I have. Let’s just say we keep the storage fibre links on a dedicated switch. With that said, each link is going to require it’s own IP - I’m assuming each virtual host will require an IP address for it’s fibre card and at the other end of this, the fibre link(s) on the NAS are also going to require IPs. Is it typical in this scenario to use hard-coded IPs or would you still use DHCP in a setup like this? I’m sorry, I have never done this before. Just a quickest I was thinking about!

Thanks again for everyone’s help and input!

20

Static IPs.

and for the Storage server you would be using interface bonding /LAPC and also tell the switch that these ports where the storage is directly connected use LAPC.

All these direct connections to the storage would be addressed by a single IP address by the client hosts.