So I am looking at replacing a few old servers with XCP and Freenas. I have setup my XCP server (specs below) and Freenas using both NFS and ISCSI over 10gb. And while I have been able to improve my write speeds on the VM to much faster than the current phyical machine is writing to its own RAID 5, my read speeds are suffering. The wall of text below is my setup. (This thread will be updated with actual perfomance data, but basically the Physical server is getting ~300mb p/s read vs ~200 on the VM)
Just wanted to see if there was anything glaring that I am missing or doing wrong. Happy Sunday!
Dell R620
2xE5-2680v2 CPUs
192GB Of RAM
82599ES 10GB NIC Card (MTU 9000)
FreeNAS Server
Summary
Dell C2100
72 GB of RAM
2x Xeon E5645 CPUs
12x 2TB 7200RPM SAS Drives (Two 5 Drive VDevs in a RAID2Z configuration, with two Hot Spares)
1x 500gb SSD for additional Cache, likely overkill, but I do see it being used
1x 64gb SSD for boot
10GB NIC (MTU 9000)
Been a while since I have dug into the auto-tuning option, but I don’t think it is needed with the newer versions of FreeNAS unless you have a special use case such as not having much memory which is not the case for your setup.
@LTS_Tom - That was my understanding as well, but I figured it couldn’t hurt.
@FredFerrell - I dont believe this to be the issue. In the Physical server, I am using a dell H200 with 3Gbps drives, vs the H310 with 6gbps drives in the Freenas box.
I have verified with iperf that I am getting ~8.8 gpbs between Freenas and Xenserver, and the simply using DD comand, the speeds are much more in line with what I would expect.
Summary
Write Test to NFS SR
[08:23 xcp-Nebula ee8e20ee-93ea-9dc6-c62c-937a35b4d2dd]# dd if=/dev/zero of=./test1.img bs=2GB count=1 oflag=dsync
I don’t expect your cache to be beneficial in your reads since the data being read is a one time thing. Cache for reads is only useful when you have “hot” data being accessed multiple times. Now on the writes it would be.
Agreed, not sure if you saw my edits to my 2nd post, but this is clearly a tuning thing I suspect. When I simply run dd commands from the Xen server to the Freenas SR speeds are much more inline with what they should be.
Regarding the HBAs, in my personal experience those are not that good. I have built some decent performing storage using LSI 9207-8i HBAs. They are easy and cheap to get on eBay.
What are the hard drives you are using on the physical box and how many in the RAID array?
What dd commands are you running. Do they read/write data to the disks?
For the Write Test, I ran the following inside of the NFS mount point (the Output file test1.img is being written to Freenas from the Xen server) dd if=/dev/zero of=./test1.img bs=2GB count=1 oflag=dsync
It completed at 303 MB/s
For the read test, I read that file I just created from Freenas to the XenServer dd if=./test1.img of=/dev/null bs=8k
It completed at 654 MB/s
Regarding the HBAs, in my personal experience those are not that good. I have built some decent performing storage using LSI 9207-8i HBAs. They are easy and cheap to get on eBay.
What are the hard drives you are using on the physical box and how many in the RAID array?
So, I am not super educated on HBA’s and the entire role, but if I am seeing good performance on the Freenas itself, as well as expected performance from the array using DD I am not sure that the HBA can be blamed? The physical box is using 1TB 7200RPM 3GB/s
I am working on confirming the layout of the Physical box, but I am not sure that matters as writing from the Xenserver to FreeNAS is performing quite well, its simply inside the VM that is not.
Just thought of something too. You should check the HBA BIOS to see if caching is turned on.
Again, not sure the HBA matters as I am getting good numbers outside of the VM’s
Also, just to throw it out there, the current testing VM is a Win10 box, using the PV Storage Driver from Citrix, v 9.0.0.30 (released on 10/11/2019)
Is there any chance that the file used for the read test is on the SSD cache and not the spinning disks? If it was just written prior to the read test this would probably be the case.
I did not mean to walk away from this thread, I have been traveling. I will be doing more testing today based on your suggestions and report back!
And just to be clear, the performance on the Pool itself has been as expected, its only when using the pool for VMs that it suffers.
Update:
Using a mirrored RAID2Z array with a SSD Log drive, Read performance is still ‘slow’ at 224 MB/s compared to the 348MB/s on the physical server.
@FredFerrell - Do you mean on the physical server? The physical server has no caching, just a RAID5 array on 3gb/s SATA drives. If you mean on the VM, I hope that the 224 mb/s are not coming from the cache drive, that would be disappointing.
@sdfungi - This result is from 2, 5 drive RAIDz2 VDEVS in mirror, with both Sync and ATIME off. I will rebuild the pool using simply Raidz and post results.
Here is a good speed test from the same hardware as my XCP server, but running Windows and simply mounting the freenas via ISCSI. I think there is some tuning to be done, but this looks good.
Now here is a VM running on identical hardware, same Freenas Pool, but mounted over NFS (I will do more testing, but i dont yet think this is a ISCSI v NFS issue) Below are some Pool/ZVOL/DATASET settings
Phy Box with ISCSI LUN
- Pool Config
-3x2 TB Drives in RAIDZ1
- Pool Settings
-Sync = Standard
-lz4
-atime=on
-dedup=off
-recordsize 128k
-Zvol
-Sync=standard
-iscsi/extents
-Block size =512
-lun rpm=7200
VM Running on NFS
- Pool Config
-3x2 TB Drives in RAIDZ1
- Pool
-Sync = Standard
-lz4
-atime=on
-dedup=off
-recordsize 128k
-Dataset
-Sync=Disabled
-Compression=lz4
-Atime=On
-Dedupe=Off
-Recordsize=128k
OK! Can confirm, no matter NFS, ISCSI, MIrror VDEVs, RAIDZ, etc, I am hitting a wall at ~400MB/s. (Or about 3gb/s) This is confirmed by my iperf test from the Windows 10 VM to the Freenas box.
So now the question is, why would the Host machine get +8.9gb/s while the guest VM’s, even at a storage level, only be getting the 3gb/s over the same exact link.
