Slow READING VM data from NFS mount in XCP-NG

waveworm · January 18, 2024, 12:41pm

Hi, I have the following setup.

3x Ace Magic mini PCs all runing XCP-NG 8.3 latest. 2.5 gb network setup for storage to TrueNAS and Synology 10.199.199.31-33
I have a storage VLAN setup to each
I have a TrueNAS (10.199.199.2) server running 2 NVME drives in MIrror with 128gb of ram
Synology DS1621+ with NVME Cache with 10gb network 10.199.199.1

the XCP-ng hosts are setup with the NFS share to both Synology and TrueNAS for testing. all using storage network and all can communicate fine.

I can run iperf from each host to each other and on the 2.5 gb connections I can get full situation.
Also, I have tested moving a 5gb file from the hosts to each storage over NFS and can get full speeds, its when I READ/pull the file back is when I get slow speeds. This also affects booting of the VMs.

I have tested with other NFS clients and can get 500-800MB reading from Truenas and 200-300 from Synology.

I have setup a VM on the storage network and did iperf to and from that VM to each storage and can fully saturated the 2.5 gb network card in the ACE machines.

so in summery, each of my XCP-NG hosts use the 2.5gb nic on the storage VLAN. I can Write to my storage at full speed, just the reading from the storage is around 40-50MBPs

waveworm · January 18, 2024, 2:13pm

THis is what I get when I run a diskMark. I know that its not a perfect tool for this, but it shows what I am dealing with. Writing to the Drives is no proven. its reading from them.

Now I just setup another XCP-ng host with an older intel XEON using the same SRs and I could fully satuatre the connections - This is a 10gbe nic in these, so I know its 4 times the speed, but the same VM and SR.

So it seems like its something with NFS reading on these little Ryzen boxes. its the Ryzen 7 with 64gb ram.

waveworm · January 18, 2024, 6:47pm

I joined the older Xeon to the pool and used the same SRs and same network configs and can get full throughput on the NFS share sending and receiving. so at this poiint it is something on each of the Ace Magic hosts with the I226 intel card. it just wont read faster than around 60MBs. I can write back to the array at full speed, but not read.

LTS_Tom · January 19, 2024, 1:09pm

Check the logs on ace magic XCP-NG host for errors

waveworm · January 19, 2024, 9:34pm

Good Idea…any thoughts on what one? I assume you mean in /var/log on each node? or would it be in XOA?

LTS_Tom · January 19, 2024, 11:30pm

Yes, start with /var/log/xenstored-access.log

waveworm · January 20, 2024, 3:13am

I have 3 Ace nodes and 1 Xeon node that has the 10gb card. the Xeon does not have much in logs, but each of my three have about 30 log files of the xenstored-access. all filled with the following.

Jan 19 22:03:49 xcp-ace2 oxenstored: A14 Jan 19 22:03:49 xcp-ace2 oxenstored: A5 Jan 19 22:04:30 xcp-ace2 oxenstored: D6 Jan 19 22:04:30 xcp-ace2 oxenstored: A6 Jan 19 22:04:30 xcp-ace2 oxenstored: D6 Jan 19 22:04:30 xcp-ace2 oxenstored: A14 Jan 19 22:04:30 xcp-ace2 oxenstored: A5 Jan 19 22:04:49 xcp-ace2 oxenstored: D7 Jan 19 22:04:49 xcp-ace2 oxenstored: A6 Jan 19 22:04:49 xcp-ace2 oxenstored: D7 Jan 19 22:04:49 xcp-ace2 oxenstored: A14 Jan 19 22:04:49 xcp-ace2 oxenstored: A5 Jan 19 22:05:30 xcp-ace2 oxenstored: D6 Jan 19 22:05:30 xcp-ace2 oxenstored: A14 Jan 19 22:05:30 xcp-ace2 oxenstored: A5 Jan 19 22:05:49 xcp-ace2 oxenstored: D7 Jan 19 22:05:49 xcp-ace2 oxenstored: A14 Jan 19 22:05:49 xcp-ace2 oxenstored: A5 Jan 19 22:06:30 xcp-ace2 oxenstored: D6 Jan 19 22:06:30 xcp-ace2 oxenstored: A6 Jan 19 22:06:30 xcp-ace2 oxenstored: D6 Jan 19 22:06:30 xcp-ace2 oxenstored: A14 Jan 19 22:06:30 xcp-ace2 oxenstored: A5 Jan 19 22:06:49 xcp-ace2 oxenstored: D7 Jan 19 22:06:49 xcp-ace2 oxenstored: A6 Jan 19 22:06:49 xcp-ace2 oxenstored: D7 Jan 19 22:06:49 xcp-ace2 oxenstored: A14 Jan 19 22:06:49 xcp-ace2 oxenstored: A5 Jan 19 22:07:30 xcp-ace2 oxenstored: D6 Jan 19 22:07:30 xcp-ace2 oxenstored: A14 Jan 19 22:07:30 xcp-ace2 oxenstored: A5 Jan 19 22:07:50 xcp-ace2 oxenstored: D7 Jan 19 22:07:50 xcp-ace2 oxenstored: A14 Jan 19 22:07:50 xcp-ace2 oxenstored: A5 Jan 19 22:08:31 xcp-ace2 oxenstored: D6 Jan 19 22:08:31 xcp-ace2 oxenstored: A6 Jan 19 22:08:31 xcp-ace2 oxenstored: D6 Jan 19 22:08:31 xcp-ace2 oxenstored: A14 Jan 19 22:08:31 xcp-ace2 oxenstored: A5 Jan 19 22:08:50 xcp-ace2 oxenstored: D7 Jan 19 22:08:50 xcp-ace2 oxenstored: A6 Jan 19 22:08:50 xcp-ace2 oxenstored: D7 Jan 19 22:08:50 xcp-ace2 oxenstored: A14 Jan 19 22:08:50 xcp-ace2 oxenstored: A5 Jan 19 22:09:31 xcp-ace2 oxenstored: D6 Jan 19 22:09:31 xcp-ace2 oxenstored: A14 Jan 19 22:09:31 xcp-ace2 oxenstored: A5 Jan 19 22:09:50 xcp-ace2 oxenstored: D7 Jan 19 22:09:50 xcp-ace2 oxenstored: A14 Jan 19 22:09:50 xcp-ace2 oxenstored: A5 w event /local/domain/7/data/updated xenopsd-xc:domain-7
w event /local/domain/7/data/updated squeezed:domain-7
write data/meminfo_free 3246924
w event /local/domain/6/data/meminfo_free xcp-rrdd:domain-6
write data/updated Sat Jan 20 03:04:30 2024
w event /local/domain/6/data/updated xenopsd-xc:domain-6
w event /local/domain/6/data/updated squeezed:domain-6
write data/meminfo_free 62168
w event /local/domain/7/data/meminfo_free xcp-rrdd:domain-7
write data/updated Sat Jan 20 03:04:49 2024
w event /local/domain/7/data/updated xenopsd-xc:domain-7
w event /local/domain/7/data/updated squeezed:domain-7
write data/updated Sat Jan 20 03:05:30 2024
w event /local/domain/6/data/updated xenopsd-xc:domain-6
w event /local/domain/6/data/updated squeezed:domain-6
write data/updated Sat Jan 20 03:05:49 2024
w event /local/domain/7/data/updated xenopsd-xc:domain-7
w event /local/domain/7/data/updated squeezed:domain-7
write data/meminfo_free 3258260
w event /local/domain/6/data/meminfo_free xcp-rrdd:domain-6
write data/updated Sat Jan 20 03:06:30 2024
w event /local/domain/6/data/updated xenopsd-xc:domain-6
w event /local/domain/6/data/updated squeezed:domain-6
write data/meminfo_free 117728
w event /local/domain/7/data/meminfo_free xcp-rrdd:domain-7
write data/updated Sat Jan 20 03:06:49 2024
w event /local/domain/7/data/updated xenopsd-xc:domain-7
w event /local/domain/7/data/updated squeezed:domain-7
write data/updated Sat Jan 20 03:07:30 2024
w event /local/domain/6/data/updated xenopsd-xc:domain-6
w event /local/domain/6/data/updated squeezed:domain-6
write data/updated Sat Jan 20 03:07:50 2024
w event /local/domain/7/data/updated xenopsd-xc:domain-7
w event /local/domain/7/data/updated squeezed:domain-7
write data/meminfo_free 3244548
w event /local/domain/6/data/meminfo_free xcp-rrdd:domain-6
write data/updated Sat Jan 20 03:08:31 2024
w event /local/domain/6/data/updated xenopsd-xc:domain-6
w event /local/domain/6/data/updated squeezed:domain-6
write data/meminfo_free 121468
w event /local/domain/7/data/meminfo_free xcp-rrdd:domain-7
write data/updated Sat Jan 20 03:08:50 2024
w event /local/domain/7/data/updated xenopsd-xc:domain-7
w event /local/domain/7/data/updated squeezed:domain-7
write data/updated Sat Jan 20 03:09:31 2024
w event /local/domain/6/data/updated xenopsd-xc:domain-6
w event /local/domain/6/data/updated squeezed:domain-6
write data/updated Sat Jan 20 03:09:50 2024
w event /local/domain/7/data/updated xenopsd-xc:domain-7
w event /local/domain/7/data/updated squeezed:domain-7

waveworm · January 23, 2024, 1:09pm

I ran alll my tests and finally found that the DAC cable connecting my cluster to my storage caused the issue. it gave me full link status at 10gb, but as soon as I switched it out to a fiber connection, it worked perfectly.

so I was able to transfer at 70-80 MBs reading, and 250ish Mbs writing. as soon as I changed the cable, it fixed the issue.