TrueNAS Core VS TrueNAS Scale Feb 15 2023 Speed Test

Computer Hardware & Server Infrastructure Builds
1

I tested TrueNAS-13.0-U3.1 against TrueNAS-SCALE-22.12.0 by following this process:

  1. Loaded TrueNAS Core
  2. Created NFS & iSCSI Shares
  3. Connected to XCP-NG host
  4. Ran Phoronix Benchmarks from VM on XCP-NG
  5. Created SMB share
  6. Used Crystaldisk benchmark from Windows to share
  7. Did in place upgrade to TrueNAS Scale
  8. Ran Phoronix Benchmarks from the same VM on XCP-NG
  9. Used Crystaldisk benchmark from Windows to share
    Hardware AMD Ryzen 5 5600G 64GB ZFS 1 x RAIDZ2 SSD Sata Drives

Flexible IO Tester 3.29
Random Read Block Size: 1MB
MB/s > Higher Is Better
TrueNAS 13 NFS ............ 770  |=================================
TrueNAS 13 iSCSI .......... 475  |====================
TrueNAS Scale 22.12 NFS ... 1044 |=============================================
TrueNAS Scale 22.12 iSCSI . 794  |==================================

Random Read Block Size: 1MB
IOPS > Higher Is Better
TrueNAS 13 NFS ............ 766  |=================================
TrueNAS 13 iSCSI .......... 471  |====================
TrueNAS Scale 22.12 NFS ... 1041 |=============================================
TrueNAS Scale 22.12 iSCSI . 791  |==================================

Random Read Block Size: 4KB
MB/s > Higher Is Better
TrueNAS 13 NFS ............ 231 |==========================================
TrueNAS 13 iSCSI .......... 252 |==============================================
TrueNAS Scale 22.12 NFS ... 180 |=================================
TrueNAS Scale 22.12 iSCSI . 226 |=========================================

Random Read Block Size: 4KB
IOPS > Higher Is Better
TrueNAS 13 NFS ............ 59007 |========================================
TrueNAS 13 iSCSI .......... 64487 |============================================
TrueNAS Scale 22.12 NFS ... 46100 |===============================
TrueNAS Scale 22.12 iSCSI . 57967 |========================================

Random Read Block Size: 8MB
MB/s > Higher Is Better
TrueNAS 13 NFS ............ 860  |====================================
TrueNAS 13 iSCSI .......... 541  |=======================
TrueNAS Scale 22.12 NFS ... 1069 |=============================================
TrueNAS Scale 22.12 iSCSI . 815  |==================================

Random Read Block Size: 8MB
IOPS > Higher Is Better
TrueNAS 13 NFS ............ 104 |=====================================
TrueNAS 13 iSCSI .......... 64  |=======================
TrueNAS Scale 22.12 NFS ... 130 |==============================================
TrueNAS Scale 22.12 iSCSI . 98  |===================================

Random Write Block Size: 1MB
MB/s > Higher Is Better
TrueNAS 13 NFS ............ 654.00 |=============================
TrueNAS 13 iSCSI .......... 957.00 |===========================================
TrueNAS Scale 22.12 NFS ... 876.10 |=======================================
TrueNAS Scale 22.12 iSCSI . 647.94 |=============================

Random Write Block Size: 1MB
IOPS > Higher Is Better
TrueNAS 13 NFS ............ 650 |===============================
TrueNAS 13 iSCSI .......... 953 |==============================================
TrueNAS Scale 22.12 NFS ... 872 |==========================================
TrueNAS Scale 22.12 iSCSI . 644 |===============================

Random Write Block Size: 4KB
MB/s > Higher Is Better
TrueNAS 13 NFS ............ 202.00 |=======================
TrueNAS 13 iSCSI .......... 377.00 |===========================================
TrueNAS Scale 22.12 NFS ... 184.00 |=====================
TrueNAS Scale 22.12 iSCSI . 189.07 |======================

Random Write Block Size: 4KB
IOPS > Higher Is Better
TrueNAS 13 NFS ............ 51713 |========================
TrueNAS 13 iSCSI .......... 96483 |============================================
TrueNAS Scale 22.12 NFS ... 47120 |=====================
TrueNAS Scale 22.12 iSCSI . 48383 |======================

Random Write Block Size: 8MB
MB/s > Higher Is Better
TrueNAS 13 NFS ............ 680  |==============================
TrueNAS 13 iSCSI .......... 845  |=====================================
TrueNAS Scale 22.12 NFS ... 1022 |=============================================
TrueNAS Scale 22.12 iSCSI . 782  |==================================

Random Write Block Size: 8MB
IOPS > Higher Is Better
TrueNAS 13 NFS ............ 81  |==============================
TrueNAS 13 iSCSI .......... 102 |======================================
TrueNAS Scale 22.12 NFS ... 124 |==============================================
TrueNAS Scale 22.12 iSCSI . 94  |===================================

Random Read Block Size: 128KB
MB/s > Higher Is Better
TrueNAS 13 NFS ............ 459  |====================
TrueNAS 13 iSCSI .......... 467  |====================
TrueNAS Scale 22.12 NFS ... 1035 |=============================================
TrueNAS Scale 22.12 iSCSI . 791  |==================================

Random Read Block Size: 128KB
IOPS > Higher Is Better
TrueNAS 13 NFS ............ 3668 |====================
TrueNAS 13 iSCSI .......... 3735 |====================
TrueNAS Scale 22.12 NFS ... 8274 |=============================================
TrueNAS Scale 22.12 iSCSI . 6324 |==================================

Random Write Block Size: 128KB
MB/s > Higher Is Better
TrueNAS 13 NFS ............ 624  |============================
TrueNAS 13 iSCSI .......... 876  |=======================================
TrueNAS Scale 22.12 NFS ... 1007 |=============================================
TrueNAS Scale 22.12 iSCSI . 697  |===============================

Random Write Block Size: 128KB
IOPS > Higher Is Better
TrueNAS 13 NFS ............ 4992 |============================
TrueNAS 13 iSCSI .......... 7003 |=======================================
TrueNAS Scale 22.12 NFS ... 8051 |=============================================
TrueNAS Scale 22.12 iSCSI . 5573 |===============================

Sequential Read Block Size: 1MB
MB/s > Higher Is Better
TrueNAS 13 NFS ............ 979  |==========================================
TrueNAS 13 iSCSI .......... 898  |======================================
TrueNAS Scale 22.12 NFS ... 1051 |=============================================
TrueNAS Scale 22.12 iSCSI . 801  |==================================

Sequential Read Block Size: 1MB
IOPS > Higher Is Better
TrueNAS 13 NFS ............ 976  |==========================================
TrueNAS 13 iSCSI .......... 894  |======================================
TrueNAS Scale 22.12 NFS ... 1047 |=============================================
TrueNAS Scale 22.12 iSCSI . 798  |==================================

Sequential Read Block Size: 4KB
MB/s > Higher Is Better
TrueNAS 13 NFS ............ 433 |==============================================
TrueNAS 13 iSCSI .......... 383 |=========================================
TrueNAS Scale 22.12 NFS ... 369 |=======================================
TrueNAS Scale 22.12 iSCSI . 344 |=====================================

Sequential Read Block Size: 4KB
IOPS > Higher Is Better
TrueNAS 13 NFS ............ 110927 |===========================================
TrueNAS 13 iSCSI .......... 98107  |======================================
TrueNAS Scale 22.12 NFS ... 94533  |=====================================
TrueNAS Scale 22.12 iSCSI . 88200  |==================================

Sequential Read Block Size: 8MB
MB/s > Higher Is Better
TrueNAS 13 NFS ............ 1012 |==========================================
TrueNAS 13 iSCSI .......... 522  |======================
TrueNAS Scale 22.12 NFS ... 1077 |=============================================
TrueNAS Scale 22.12 iSCSI . 810  |==================================

Sequential Read Block Size: 8MB
IOPS > Higher Is Better
TrueNAS 13 NFS ............ 123 |===========================================
TrueNAS 13 iSCSI .......... 62  |======================
TrueNAS Scale 22.12 NFS ... 131 |==============================================
TrueNAS Scale 22.12 iSCSI . 98  |==================================

Sequential Write Block Size: 1MB
MB/s > Higher Is Better
TrueNAS 13 NFS ............ 669 |===============================
TrueNAS 13 iSCSI .......... 919 |==========================================
TrueNAS Scale 22.12 NFS ... 995 |==============================================
TrueNAS Scale 22.12 iSCSI . 713 |=================================

Sequential Write Block Size: 1MB
IOPS > Higher Is Better
TrueNAS 13 NFS ............ 665 |===============================
TrueNAS 13 iSCSI .......... 915 |==========================================
TrueNAS Scale 22.12 NFS ... 991 |==============================================
TrueNAS Scale 22.12 iSCSI . 710 |=================================

Sequential Write Block Size: 4KB
MB/s > Higher Is Better
TrueNAS 13 NFS ............ 308 |======================
TrueNAS 13 iSCSI .......... 635 |==============================================
TrueNAS Scale 22.12 NFS ... 366 |===========================
TrueNAS Scale 22.12 iSCSI . 386 |============================

Sequential Write Block Size: 4KB
IOPS > Higher Is Better
TrueNAS 13 NFS ............ 78933  |=====================
TrueNAS 13 iSCSI .......... 162667 |===========================================
TrueNAS Scale 22.12 NFS ... 93700  |=========================
TrueNAS Scale 22.12 iSCSI . 98733  |==========================

Sequential Write Block Size: 8MB
MB/s > Higher Is Better
TrueNAS 13 NFS ............ 683 |=================================
TrueNAS 13 iSCSI .......... 964 |==============================================
TrueNAS Scale 22.12 NFS ... 949 |=============================================
TrueNAS Scale 22.12 iSCSI . 697 |=================================

Sequential Write Block Size: 8MB
IOPS > Higher Is Better
TrueNAS 13 NFS ............ 82  |================================
TrueNAS 13 iSCSI .......... 117 |==============================================
TrueNAS Scale 22.12 NFS ... 115 |=============================================
TrueNAS Scale 22.12 iSCSI . 83  |=================================

Sequential Read Block Size: 128KB
MB/s > Higher Is Better
TrueNAS 13 NFS ............ 977  |==========================================
TrueNAS 13 iSCSI .......... 589  |=========================
TrueNAS Scale 22.12 NFS ... 1050 |=============================================
TrueNAS Scale 22.12 iSCSI . 789  |==================================

Sequential Read Block Size: 128KB
IOPS > Higher Is Better
TrueNAS 13 NFS ............ 7810 |==========================================
TrueNAS 13 iSCSI .......... 4706 |=========================
TrueNAS Scale 22.12 NFS ... 8401 |=============================================
TrueNAS Scale 22.12 iSCSI . 6310 |==================================

Sequential Write Block Size: 128KB
MB/s > Higher Is Better
TrueNAS 13 NFS ............ 656.0 |==============================
TrueNAS 13 iSCSI .......... 738.4 |==================================
TrueNAS Scale 22.12 NFS ... 968.0 |============================================
TrueNAS Scale 22.12 iSCSI . 738.0 |==================================

Sequential Write Block Size: 128KB
IOPS > Higher Is Better
TrueNAS 13 NFS ............ 5247 |===============================
TrueNAS 13 iSCSI .......... 5903 |==================================
TrueNAS Scale 22.12 NFS ... 7741 |=============================================
TrueNAS Scale 22.12 iSCSI . 5902 |==================================

image
image1113×475 168 KB

image
image1083×461 181 KB

Phoronix page link

1 Like
2

Thanks. There are some gains to be had with Scale, the only place that was slower was 4k.

Also shows that NFS is still a good choice. Can we assume that you turn NFS sync off like in previous tests?

3

Yes, SYNC was off for all tests.

4

I did some tests of my own to determine if it is worth migrating to scale for my use case. Sharing in case anybody is interested.
Both Core and Scale were running on proxmox (odroid h3+ with 64gb ram) with same settings (2 core, 16GB ram), data are on local SSD, which is mirrored zfs. Yea, zfs on zfs test :slight_smile:
Added my real NAS (Core HW) for comparasion, made sure disk was mostly idle at the time. Supermicro system, 128GB ECC ram, SSD stripped mirrors.
Dataset settings: Disabled sync, disabled compression, atime is off 512kb record size.
Unity 2019.4.31f1 project I was opening over SMB was 2.5GB

        1st  2nd  3rd

Local sys: 00:52, 00:25, 00:23
Core HW: 03:04, 03:07, 03:10
Core VM: 02:37, 02:27, 02:24
Scale VM: 02:08, 02:07, 02:02

I find it quite shocking that Core running on hardware directly which is much faster and more capable in every way then my proxmox server got worse results.

Still far from usable even for archive, since this is really small project, but, Scale is noticably faster for this workload.

5

Hi Tom,
I’m moving to TrueNAS Core from my long time running FreeBSD NAS. My primary use case is SMB for serving files to a few people on the network - mostly larger media files and NFS to VMs for bulk flash storage. I do not use iSCSI that often. I value how reliable and stable FreeBSD has been - and with Scale being newer I worry about switching to it for this new build for better NFS performance. Thoughts? Thank you

6

Most NFS VM use will be small writes and the “Random Write Block Size: 4KB” is barely any different between the two.

1 Like
7

Thank you. After labbing it a bit - I’m satisified with my requirements for performance on Core and have seen how the option to move to Scale works. Until next time!