CIFS vs NFS on TrueNAS Scale

1

Im currently “benching” a ZFS Pool with the LTS FIO Script and i stumbled on a stange trait. The benches over NFS where way way worse (for some scenarios like a 10%) than over CIFS. No changes whre mady between the runs.

Random Read Random Write Write Read Read/Write Time
Protokoll Options IOPS Bandwidth IOPS Bandwidth IOPS Bandwidth IOPS Bandwidth IOPS Read IOPS Write Bandwidth Read Bandwidth Write Time
cifs 64k 1g 1,259.71 78.73 8,603.57 537.72 12,240.82 765.05 28,462.76 1,778.92 9,462.24 9,524.36 591.39 595.27 08:10
nfs 64k 1g 122.32 7.64 9,083.68 567.73 156.44 9.78 27,151.00 1,696.94 133.95 134.83 8.37 8.437 02:06:03
cifs 256k 4g 690.37 172.59 2,495.34 623.83 721.92 180.48 8,992.60 2,248.15 1,286.58 1,295.03 321.64 323.76 15:44
nfs 256k 4g 115.72 28.93 2,993.99 748.50 135.42 33.85 7,032.00 1,758.00 122.68 123.48 30.67 30.877 02:20:39
cifs 1m 8g 601.42 601.42 561.40 561.40 1,785.03 1,785.03 2,237.10 2,237.10 1,011.31 1,034.08 1,011.31 1,034.08 17:04
nfs 1m 8g 112.03 112.03 462.91 462.91 128.33 128.33 1,852.01 1,852.01 115.66 118.27 115.66 118.277 01:21:10
cifs 4m 16g 141.37 565.49 105.92 423.67 428.59 1,714.35 534.77 2,139.08 240.02 247.49 960.06 989.96 39:07
nfs 4m 16g 57.12 228.49 79.42 317.67 79.37 317.46 474.62 1,898.49 55.36 57.09 221.45 228.357 01:31:57

Why is my bench so mutch worse over NFS than CIFS. The bench was running over an virtual ~25gbit link from Proxmox to an virtual TrueNAS.

What did i miss?

2

Did you disable sync on the dataset?

3

The only change ive made was the target of the bench.
/mnt/TNS-NFS vs /mnt/TNS-CIFS

Sync option from the Dataset is standard
Options from the mount:

mount -t nfs IP:/mnt/pool/dataset /mnt/TNS-NFS
mount -t cifs -o username=user-test1,password=user-test1 //IP/Dataset /mnt/TNS-CIFS
4

What I am saying is NFS takes a hit when using synchronous writes when using ZFS.

2 Likes
5

Thats an behavior that i dindt expect. But is it worth it to use NFS when you have to disable Sync? NFS is a bit faster but why take the hit and disable sync?

I have run a 64k 1g test over NFS with Sync disabled and now its comareable to CIFS.

6

Ive now run a bench with cifs and sync=always, with the same bad result as nfs with sycn=standard.

So it seems that cifs dose not request the sync feature and nfs dose.
But since im not planning to run VMs or databases from this pool there shouldt be a reason to not disable sync? Whats the drawback since cifs seems to allready not use it?