I need to try and budget for a couple new storage servers, and what I’m seeing on Supermicro’s shop is that the closest chassis to what I need is running an AMD EPYC™ 7313 Processor 16-Core 3.00GHz, going to go with 2x32 ram and a 10gb Intel card.
Not sure if I need a storage controller or not yet.
Going to go with dual system drives in the back, and 6 storage drives in the front, giving room to grow. Function is going to be storage for VM and I don’t need a ton of storage for this (6x960). All SSD this time, maybe I can add some spinning drives if I need bigger and slower storage, or just grow the set with more drives.
I’d say that would be plenty of power. I would recommend installing an optane drive for ZIL. It’s got good safety in case all power is lost then when the server comes back up it will complete the writes. And will have better performance on writes.
Other than that I’d say you have a decent setup. You can never go wrong with more RAM though if you are needing more read cache.
What SSD are you planning on using for your flash pool?
I’m not really sure what brand SSD they sell, but they are generally very good. My only concern is they are expressed in writes per day, and due to money I’m going to need to select a 1.5x version. Then I thought about it, and there is zero percent chance that I average 3TB per day on the array or even 960GB per day on a single drive. The VMs that I run just don’t have that many writes.
And the two OS drives certainly won’t have a lot either.
I’ll have to look into an Optane for each, I’m surprised they didn’t show this as an option. Should I size the Optane to be as big as the RAM (64GB) or some other size? The last time I messed with any type of cache, I was using slow drives and the RAM cache was faster, so I stopped (current hardware)
The board looks like it has two nvme m.2 sockets, plus being a 2u chassis there’s room to use a PCIe x16 slot or two, I’ll look into this.
Thanks. Waiting for ixsystems to get back to me on what they can provide that’s similar, sometimes buying an “appliance” is easier than buying a server, especially if it goes over a magic $5000 to become “equipment”.
Sometimes the budgeting and purchasing hoops are just too much to jump through. And other times I’ve bought a case, mainboard, RAM, processor, etc. as separate items because it was the only way to get what we needed through the process.
If this is for a company I’d highly recommend getting an appliance and a support contract. That way you have some leverage with ixsystems when things go south. It would give you and management better sleep at night.
This server is very good, I have used some of this line at OVH and it will certainly not let you down, quite the opposite.
Some points I would make if I were you:
1 - I would install 2 NVMe enterprise drives in mirror for SLOG/Zil (write cache), having only 1 disk for slog is complicated, because if it dies, there will be data loss;
2 - I would install more memory if possible, or plan to increase it to 128GB of RAM, to have better read performance, because there will be much more data in RAM cache and this can be interesting for virtualization;
I am not an expert in TrueNAS, on the contrary, I am even looking for courses on it, but I got these 2 tips from my experience in a large project I worked on, with a ZFS expert, I was going to set up a TrueNAS with only 1 disk for SLOG and he showed me the headache that would arise with the loss of the log disk in the middle of the process, we had a Proxmox VM corrupted during this loss in the application of Windows Update (we caught a critical moment to test).
I just priced it out and put two of these drives in:
7450 MAX NVMe PCIe 4.0 Solid State Drive
These are Micron drives and seem to be fairly fast, the MAX gives better drive writes per day which for a cache might be important. I also lowered my expectations and went with spinning drives and a better NIC. Spinning drives made about a $400 price difference. The NVME were about $400, and the better NIC was only about $40 additional and made sense to move forward from Intel X520 to Intel X710. My XCP-NG hosts are X710 and working fine with DAC cables.
I did leave room for more RAM, only installing 2x32GB and I think it said 8 sockets, so plenty of room for a storage only server. I’m not seeing a lot of usage on my lab system which has 96GB of ram, it never seems to take all of the free RAM to use a ZFS cache, highest I’ve gotten is 64GB leaving around 20GB as free. But I also think I need more than 32GB, and need at least a pair of modules.
I’m also talking to ixsystems, but it looks like all I’m going to be able to afford are Mini-R which is pretty close to the same specs as the Supermicro.
Call it brand loyalty, or something else, every server in my racks is a Supermicro. I’ve priced Dell and HP, and always come back to Supermicro being just as reliable and less money. They are also easy to get here in the USA.
[edit] sent my specs to 45drives to see what they can offer.
I believe this is a good choice! These NVMes for SLOG will help a lot in writing, which is the most difficult bottleneck to solve on a server.
In this case, what type of systems are you going to run on this server? I ask this because in some cases, even with a RAIDZ with multiple disks, replacing an SSD with an HD can result in a huge loss of performance.
For example, in a database, the time for a query on HD and SSD, even with both having similar sequential read and write rates, when you calculate the query execution time, the difference can be quite large.
Perhaps, if you can’t afford it now, it’s worth setting up the RAIDZ without the SLOG and adding it in 1 or 2 years, when you have the money for it. The process of adding the SLOG is simple and can be done with the volumes already created; you just need to turn off the TrueNAS to plug in the disks.
I may need to set this up with HDD and then upgrade to SSD in the future. We don’t have any real databases that need high speeds right now, and probably never going to have that need for me. The biggest database that gets referenced is my antivirus controller (Trellix ePO) and that’s not really a concern. DHCP, DNS, that stuff seems to be fine on the spinning drives that I’m using now. OS updates on the Windows servers are the biggest pain right now and one of the reasons I want to go to faster storage. I’d also like to reduce power draw and fan noise if possible, and SSD should help me get there.
I may need to piece meal this build, and maybe even want to do that with a 45 drives “homelab” chassis (still built on Supermicro main board). Just got off the phone with Josh from 45drives and need to look into some things.
Windows Update is really a bomb, or rather, Windows is a bomb for disks… Using Windows on a HDD is a pain. It’s possible to use it, but nowadays when I need to use a HDD, I know that when I need it most I feel like setting the server on fire. hehehehe
In terms of noise, the HDD is louder, but it’s much smaller than the coolers on a rack-mount server, so the difference won’t be that big, I think.