Disparity between blackmagic speed test and file copy speed

Networking & Firewalls
1

I have a TrueNAS VM on Proxmox with a passed through ConnectX 3 card. It’s passed through as a RAW device, with PCI-e checked in Proxmox.

Also passed through to the TrueNAS VM is a 1TB nvme, again, PCI-E, raw device.

I’ve an iMac with a thunderbolt ten gig adaptor connecting to it.

Both devices are connect to a Unifi Pro XG 10GB switch - the TrueNAS Connect X via DAC, the iMac via RJ45.

I’ve noticed that no matter what, when I cp or rsync a file from the iMac to the nvme SMB share, I cannot exceed 250MB/s writing to the nvme.

My rsync command is --whole-file --progress with a 1GB, 4GB and 8GB test files - all ISOs so hopefully one contiguous file to give the best possible or at least consistent outcomes.

However, using Blackmagic disk speed test I can write to the same nvme share at about 700MB/s and read at 900MB/s using the 1GB file size. Iperf3 shows a consistent 9.8Gb/s connection.

I’ve another Linux Mint VM (again, inside Proxmox) on a 10GB connectX3 in a Thinkcentre M920 tiny and that also cannot gcp the same files (as cp has no way that I know of to output progress) any faster than 200MB/s.

It is the disparity between Blackmagic and rsync/gcp I don’t understand. Is there a setting I have misunderstood? I’ve even turned off sync writes (this is a test dataset, not production, nothing’s on there I need or want to keep) to experiment but that made no difference.

I appreciate there’s a lot of layers, media and protocols involved and I wouldn’t expect ‘line speed’ but I don’t understand why the manual file copies seemingly won’t exceed 250MB/s yet Blackmagic, which reads and writes a file returns such faster speeds. If it were rsync then why is gcp also limited to 200/250MB/s? If it were MacOS, then why is Mint also returning the same result?

I even tried rclone copy - same speed limitation.

All advice welcome, happy to provide any data requested.

2

While I am sure it’s not the answer you want to hear, these are the type of issues I have run into when people virtualize TrueNAS. Have you tried using iperf to see what speed you can achieve with raw networking?

3

You raise a very good point and the ‘inconsistency’ is certainly affected by it.

iPerf3 reports the full 10GB speed:

onnecting to host truenas, port 5201
[ 6] local 192.168.1.41 port 51604 connected to 192.168.1.10 port 5201
[ ID] Interval Transfer Bandwidth
[ 6] 0.00-1.00 sec 1.03 GBytes 8.85 Gbits/sec
[ 6] 1.00-2.00 sec 1.09 GBytes 9.37 Gbits/sec
[ 6] 2.00-3.00 sec 1.09 GBytes 9.37 Gbits/sec
[ 6] 3.00-4.00 sec 1.09 GBytes 9.38 Gbits/sec
[ 6] 4.00-5.00 sec 1.09 GBytes 9.37 Gbits/sec
[ 6] 5.00-6.00 sec 1.09 GBytes 9.39 Gbits/sec
[ 6] 6.00-7.00 sec 1.09 GBytes 9.34 Gbits/sec
[ 6] 7.00-8.00 sec 1.09 GBytes 9.37 Gbits/sec
[ 6] 8.00-9.00 sec 1.09 GBytes 9.38 Gbits/sec
[ 6] 9.00-10.00 sec 1.09 GBytes 9.38 Gbits/sec

[ ID] Interval Transfer Bandwidth
[ 6] 0.00-10.00 sec 10.9 GBytes 9.32 Gbits/sec sender
[ 6] 0.00-10.00 sec 10.8 GBytes 9.32 Gbits/sec receiver

4

Just to mention, I did revert to a bare install. While cp is almost wirespeed rsync, rclone are still seemingly locked at 250MB/s writing from nvme to nvme over 10GB gigabit.

5

Blackmagic writes internally supplied test data to a temporary/test file on the selected target, bypassing the need to read an existing source file from the iMac’s storage. While I am not 100% sure, it is possible the data is written straight from memory or read back into memory. What kind of checksumming it uses, if any, I am not sure.

Rsync does checksumming. While I do not know your CPU speeds, it does checksums which at that speed could add some (perhaps significant) bottleneck. Try this:

rsync --checksum-choice=none --progress source.iso /mnt/share/

If your rsync doesn’t support checksum-choice=none, look for the least CPU intensive checksumming.

I have a virtualized TrueNAS running for several years without any issues, with a 10G ethernet card - though not passthrough, it is native to Proxmox and part of a virtual switch, and a virtual 10G NIC for the TrueNAS VM - and a passthrough HBA to TrueNAS with 4 SSDs, and I get near wire speeds in file transfers from my computer.

6

Hullo all, DrHeat - thank you, I am getting far closer wire speeds using the checksum-choice option.

A cp is still faster as, I imagine; it doesn’t do that so for the big copy I’ll cp, then follow up with rsync to verify (that’s a separate test!).

The CPUs are intels, not super fast ones - the Core models rather than the newest err, Core? Intel really need to use better product names.

You’re right on the virtualisation. I’ve rebuilt at the mo but when I do some maintenance (the array is 2 striped, 4 disk raidz1’s and I’ll prob move that to striped mirrors) I’ll virtualise it again.

Again, thanks all!

7

Glad it worked out!

If I may suggest, do not passthrough the 10G adapter. Instead keep it native to Proxmox and create a virtual switch. Then create a virtual network adapter for TrueNAS. This has several benefits we can get into more if you’re interested.