Slow transfer/network speeds

Networking & Firewalls
1

First off I would like to thank everyone who takes the time to even start reading this, and a bigger thanks to those who may be willing to help.

Brief intro before the details.
On my Plex server, which is being used for Plex and Handbrake. I have a script that ingests a video file, typically 4-16Gb in size, processes it and outputs a file about half the size. Within the bash script the files, respectively, transfer using cp “$file” /media/edit/orig/"$filename" for the ingest and cp /media/edit/process/"$filename" /media/Processed/"$dirname"/"$filename" for the completed transfer. “/media/edit/orig” resides on mirrored NVME drives, “/media/Processed/” resides on 8 drive raidz2 Via an NFS mount. The original files reside on a Truenas box connected to the Plex server Via a NFS mount.

mounts

┌─[root@plex]─[~]
└──╼ #df -h
Filesystem Size Used Avail Use% Mounted on
udev 16G 0 16G 0% /dev
tmpfs 3.2G 3.0M 3.2G 1% /run
/dev/sdi2 78G 6.0G 68G 9% /
tmpfs 16G 4.0K 16G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sdi1 511M 3.5M 508M 1% /boot/efi
PlexDB 703G 128K 703G 1% /PlexDB
PlexDB/PlexDB 764G 62G 703G 9% /var/lib/plexmediaserver/Library/Application Support/Plex Media Server
PlexDB/movie_edit 709G 6.3G 703G 1% /media/edit
192.168.88.2:/mnt/EsSandT/ISO/processed/ 7.1T 3.4T 3.7T 48% /media/Processed
192.168.88.2:/mnt/EsSandT/ISO/originals/ 7.1T 3.4T 3.7T 48% /media/Original
tmpfs 3.2G 0 3.2G 0% /run/user/0

My curiosity/irritation… the transfer of the original file and the subsequent transfer of the completed file transfers at about 25-30Mbs over a 10G DAC NFS share (taking nearly 20 minutes each). Where from my laptop I get 135Mbs through a 1G SMB share (claiming about 2 minutes).

What can I check to try ad figure out the bottleneck and ultimately correct it.

The details…

Thge Plex server has a Mellanox Technologies MT26448 (HP 10Gbe) → MIKROTIK CRS305-1G-4S+ → TrueNAS with a emulex corporation oneconnect 10gb nic (HP | 614203-B21 | NC552SFP | 10Gb)
All are connected with SFP+ DAC cables from 10Gtek.

All network settings, aside from address and subnet, are at default settings.

iperf Plex to NAS

root@nas:~ # iperf -s

Server listening on TCP port 5001
TCP window size: 128 KByte (default)

[ 4] local 192.168.88.2 port 5001 connected with 192.168.88.3 port 52454
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 7.25 GBytes 6.22 Gbits/sec
[ 5] local 192.168.88.2 port 5001 connected with 192.168.88.3 port 52456
[ 5] 0.0-10.0 sec 7.12 GBytes 6.12 Gbits/sec

iperf NAS to Plex

┌─[root@plex]─[~]
└──╼ #iperf -s

Server listening on TCP port 5001
TCP window size: 128 KByte (default)

[ 4] local 192.168.88.3 port 5001 connected with 192.168.88.2 port 56107
[ ID] Interval Transfer Bandwidth
[ 4] 0.0000-10.0166 sec 10.8 GBytes 9.23 Gbits/sec
[ 5] local 192.168.88.3 port 5001 connected with 192.168.88.2 port 42059
[ ID] Interval Transfer Bandwidth
[ 5] 0.0000-10.0045 sec 10.8 GBytes 9.30 Gbits/sec

dd from Plex to NAS Via NSF mount

┌─[✗]─[root@plex]─[~]
└──╼ #dd if=/dev/urandom of=/media/Original/zero.img bs=1024M count=10 status=progress iflag=fullblock
10737418240 bytes (11 GB, 10 GiB) copied, 197 s, 54.4 MB/s
10+0 records in
10+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 197.218 s, 54.4 MB/s

iftop

1.86Gb 3.73Gb 5.59Gb 7.45Gb 9.31Gb
mqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqq
192.168.88.3 => 192.168.88.2 31.1Mb 25.7Mb 25.4Mb
<= 159Kb 129Kb 128Kb
192.168.88.3 => 192.168.88.15 0b 157b 157b
<= 0b 0b 0b
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
TX: cum: 40.2GB peak: 31.1Mb rates: 31.1Mb 25.7Mb 25.4Mb
RX: 64.4GB 159Kb 159Kb 129Kb 128Kb
TOTAL: 105GB 31.2Mb 31.2Mb 25.8Mb 25.5Mb

Plex storage

┌─[root@plex]─[~]
└──╼ #zpool status PlexDB
pool: PlexDB
state: ONLINE
scan: scrub repaired 0B in 00:04:26 with 0 errors on Sun Oct 10 00:28:32 2021
config:

    NAME                                             STATE     READ WRITE CKSUM
    PlexDB                                           ONLINE       0     0     0
      mirror-0                                       ONLINE       0     0     0
        nvme-SPCC_M.2_PCIe_SSD_7FBC070B0DD900014101  ONLINE       0     0     0
        nvme-SPCC_M.2_PCIe_SSD_7FBC070B0DD900014024  ONLINE       0     0     0

errors: No known data errors

NAS storage

root@nas:~ # zpool status EsSandT
pool: EsSandT
state: ONLINE
scan: scrub repaired 0B in 12:40:01 with 0 errors on Sun Oct 17 12:40:04 2021
config:

    NAME                                            STATE     READ WRITE CKSUM
    EsSandT                                         ONLINE       0     0     0
      raidz2-0                                      ONLINE       0     0     0
        gptid/7ed9aed7-9c8e-11e8-852e-bc5ff4751799  ONLINE       0     0     0
        gptid/80445b9f-9c8e-11e8-852e-bc5ff4751799  ONLINE       0     0     0
        gptid/8131a36d-9c8e-11e8-852e-bc5ff4751799  ONLINE       0     0     0
        gptid/8256e673-9c8e-11e8-852e-bc5ff4751799  ONLINE       0     0     0
        gptid/837b3825-9c8e-11e8-852e-bc5ff4751799  ONLINE       0     0     0
        gptid/8454597b-9c8e-11e8-852e-bc5ff4751799  ONLINE       0     0     0
        gptid/852e1706-9c8e-11e8-852e-bc5ff4751799  ONLINE       0     0     0
        gptid/8678b327-9c8e-11e8-852e-bc5ff4751799  ONLINE       0     0     0

errors: No known data errors

Drives are 4TB WD Reds (non shingled)

Any help or suggestions would be greatly appreciated.

Side note… I do not know why the code blocks are not displaying properly in the iperf details.

Again, Thank you for your very precious time.

Kindly,
Eric

2

Do you have SYNC turned off on the destination ZFS dataset?

3

Sync is set to standard on both machines.

4

Configure the ZFS datasets to zfs set sync=disabled and see if the speed improves.

5

Thank you. That is a considerable difference.

In your experience what would be the likely hood of file corruption in transit with sync being disabled?

6

When setting sync=disabled it forces synchronous writes to be handled as asynchronous in the ZFS pool but tells the application doing the request the data has been written completely to the drive. ZFS is a COW (Copy on Write) file system so there is always a copy of the previous data until the new data is committed to the disk and verified. So the risk is losing any uncommitted data that is in memory if the system were to suddenly fail.

7

That eases my mind. I thank you again, for your time. It is greatly appreciated.

1 Like