Slow VM Disk Performance (XCP-Freenas)

Computer Hardware & Server Infrastructure Builds
21

Sure! (Also, @FredFerrell thank you so much for your continued efforts in this!)

First, Host-Freenas (I could only install iperf on the Xenserver, not iperf3)

root@freenas[~]# iperf -s

Server listening on TCP port 5001
TCP window size: 256 KByte (default)

[ 4] local 193.0.0.1 port 5001 connected with 193.0.0.101 port 37824
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 10.2 GBytes 8.77 Gbits/sec

[15:30 xcp-Nebula ~]# iperf -c 193.0.0.1

Client connecting to 193.0.0.1, TCP port 5001
TCP window size: 724 KByte (default)

[ 3] local 193.0.0.101 port 37824 connected with 193.0.0.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 10.2 GBytes 8.79 Gbits/sec
[16:13 xcp-Nebula ~]#

Now from Windows VM, using exact same NIC from above

root@freenas[~]# iperf3 -s

Server listening on 5201

Accepted connection from 193.0.0.105, port 49870
[ 5] local 193.0.0.1 port 5201 connected to 193.0.0.105 port 49871
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 321 MBytes 2.69 Gbits/sec
[ 5] 1.00-2.00 sec 351 MBytes 2.95 Gbits/sec
[ 5] 2.00-3.00 sec 359 MBytes 3.01 Gbits/sec
[ 5] 3.00-4.00 sec 366 MBytes 3.07 Gbits/sec
[ 5] 4.00-5.00 sec 345 MBytes 2.90 Gbits/sec
[ 5] 5.00-6.00 sec 340 MBytes 2.86 Gbits/sec
[ 5] 6.00-7.00 sec 342 MBytes 2.87 Gbits/sec
[ 5] 7.00-8.00 sec 357 MBytes 3.00 Gbits/sec
[ 5] 7.00-8.00 sec 357 MBytes 3.00 Gbits/sec

[ ID] Interval Transfer Bitrate
[ 5] 0.00-8.00 sec 2.90 GBytes 3.11 Gbits/sec receiver
iperf3: the client has terminated

Capture
Capture829×321 56.2 KB

23

I would be interested to see if a Linux VM behaved the same way as your Windows VM. Any chance you could test that?

24

SLIGHTLY better. I am finding some older threads on the Citrix forums with people experiencing similar issues, but never seemed to get resolved. This almost has to be at the hypervisor level right? (PLEASE feel free to correct me, I am just spit balling) But since the underlying OS on the Xen server (Freebsd?) is able to get the ‘full’ bandwidth, its unlikely to be any of the following:

  • Host NIC
  • Freenas NIC
  • Either DAC Cable
  • 10GB Unifi Switch

And since a physical box is able to write to the Freenas box a lot faster, I am ASSUMING that, at least right now, my performance is not my disk or layout configuration?

image

25

@Woogi, I agree with all your points. Any chance you could install a different hypervisor to see if there is a difference?

26

Sure, I have some extra hardware. This is running XCP-NG v 8.01. Actually, I think I have Server 2016 running on identical hardware. Let me see about spinning Hyper-V on it.

27

Welp, shit gets weird.

Hyper-V Windows VM (VHD is hosted on an ISCSI connection to the freenas box)
Appears to have same iperf3 speeds (approx 3gb/s), however the ATTO scores are off the charts

image
image528×644 27 KB

Freenas 10GB Interface

image
image2212×498 57.6 KB

However XCP, on the same Pool, with an additional disk added, that is hosted on the ISCSI pool

image
image530×642 21.9 KB

image
image1177×158 7.95 KB

28

I think we are seeing a difference in performance numbers for throughput due to different block sizes used in testing. On the two ATTO disk tests shown above, does the IO match for each block size tested with it?

Also, I would limit that testing from 512B to 4KB since those are what Linux and Windows use as defaults.

29

WELL…WELL…WELL, I THINK I may have found my issue:

According to Dell my C2100 has “Two, 8x PCI-E Gen two Slots”

According to Wiki
"The PCIe 2.0 standard doubles the transfer rate compared with PCIe 1.0 to 5 GT/s and the per-lane throughput rises from 250 MB/s to 500 MB/s. Consequently, a 32-lane PCIe connector (x32) can support an aggregate throughput of up to 16 GB/s. "

I will be replacing the C2100 with a R720 first thing next week and will report back.

2 Likes
30

So after thinking about it my PCI-E reasoning didnt make sense. SO, after some more testing I install CITRIX xen 8.1, and used the citrix drivers and well…MUCH better.

Phyical Server I am comparing to

image
image698×361 32.8 KB

Citrix Xen Server with Citrix branded drivers installed
image

NOW, the next step is to install XCP-NG 8.1 and try to use the Citrix branded drivers that I…aquired…from the Citrix branded Xen server.

1 Like
31

Nice work! Let me know what you find out.

32

@Woogi

I recently got a pair of HP DL360p gen8 servers, made a storage server with a 10 2TB drives with a HBA and I am running into this same 3 GB/s one way bottleneck from a XCP-NG VM to my FreeNAS server. I was wondering if you were able to figure out where the bottleneck was in your setup.

33

Are you using 10Gbit connectivity between XCP-ng and the FreeNAS server? If so, I think 3 GB is the fastest you’ll be able to get.

34

SOMEHOW I got it all working right just last night. I think it may have something to do with the VM drivers. I was using the older 7.4 Citrix drivers. I loaded the 8.1 Citrix drivers and I started getting the speeds I thought I should be getting. Although I also installed the testing drivers (on each host) from here around the same time. So I added too many variables to know which one it was.

While troubleshooting I decided enough was enough and I destroyed my entire set up (XCPNG and FreeNAS) and started from scratch. I started the the below configuration but with direct connection. I definitely seen an increase in speed so then I introduced the 10GB switch again. When I did this I noticed a slight drop in speeds. I implemented jumbo frames. This helped a lot but I was still hitting the 3-4GB/s bottleneck in write speeds to FreeNAS. This is when I found the link above and installed the newer drivers in both the hosts and on my windows VM. For right now I am happy with the speeds but over the weekend I am going to resinstall XCP-NG and try one more time to see if a fresh load without the testing drivers on my host brings the problem back.

My Hardware:

XCP-NG
HP DL360p gen8
Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
64GB RAM
Mellanox Connectx-2

Switch: Mikrotik CRS305-1G-4S+

FreeNAS
Custom build
Sabertooth 990fx r2.0
AMD FX™-8350
32GB RAM
Chelsio T320 10GbE Dual Port Adapter
LSI 9201-16i HBA
10 HGST 7200 RPM drives (5 mirrored vdevs)

From XCP-NG:
09:00.0 Ethernet controller: Chelsio Communications Inc T320 10GbE Dual Port Adapterstrong text
lspci

04:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)

ifconfig eth4

eth4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
ether 00:02:c9:51:19:74 txqueuelen 1000 (Ethernet)
RX packets 86060577 bytes 508920314236 (473.9 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 88191492 bytes 604182491033 (562.6 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ethtool -i eth4

driver: mlx4_en
version: 4.0-0
firmware-version: 2.9.1000
expansion-rom-version:
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

From FreeNAS:
lspci

09:00.0 Ethernet controller: Chelsio Communications Inc T320 10GbE Dual Port Adapter

ifconfig cxgb0

cxgb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: 10GB Offline Storage
options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:07:43:07:c6:d4
hwaddr 00:07:43:07:c6:d4
inet 10.10.10.205 netmask 0xffffff00 broadcast 10.10.10.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet 10Gbase-Twinax

This is a speed test from a Windows 10 VM over iSCSI :
image