Which (Software-)RAID for NVMe/SSD disks?

Today we ran some tests and benchmarks and while in the past with mechanical hard drives the hit from the overhead of running checksums etc for RAID5 and/or RAID6 was negligible, with SSD it became a factor and with the latest generation Optane disks the difference is just huge. Compare the following numbers taken on a 56 vCore server with 4x Intel Optane NVMe drives configured as RAID10, RAID5 and RAID0:

 Server24 56 Cores with RAID10 NVMe, 256 GB RAM - 20201128
  
 Sat Nov 28 11:04:33 CET 2020
 
 Basic System Information:
 ---------------------------------
 Processor  : Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
 CPU cores  : 56 @ 1707.405 MHz
 AES-NI     : ✔ Enabled
 VM-x/AMD-V : ✔ Enabled
 RAM        : 251.5 GiB
 Swap       : 4.0 GiB
 Disk       : 165.3 GiB
 
 fio Disk Speed Tests (Mixed R/W 50/50):
 ---------------------------------
 Block Size | 4k            (IOPS) | 64k           (IOPS)
   ------   | ---            ----  | ----           ----
 Read       | 527.42 MB/s (131.8k) | 615.91 MB/s   (9.6k)
 Write      | 528.81 MB/s (132.2k) | 619.15 MB/s   (9.6k)
 Total      | 1.05 GB/s   (264.0k) | 1.23 GB/s    (19.2k)
            |                      |
 Block Size | 512k          (IOPS) | 1m            (IOPS)
   ------   | ---            ----  | ----           ----
 Read       | 565.11 MB/s   (1.1k) | 597.10 MB/s    (583)
 Write      | 595.13 MB/s   (1.1k) | 636.87 MB/s    (621)
 Total      | 1.16 GB/s     (2.2k) | 1.23 GB/s     (1.2k)   
 
 iperf3 Network Speed Tests (IPv4):
 ---------------------------------
 Provider        | Location (Link)           | Send Speed      | Recv Speed
                 |                           |                 |
 Clouvider       | London, UK (10G)          | 888 Mbits/sec   | 650 Mbits/sec
 Online.net      | Paris, FR (10G)           | 880 Mbits/sec   | 533 Mbits/sec
 WorldStream     | The Netherlands (10G)     | 913 Mbits/sec   | 815 Mbits/sec
 Biznet          | Jakarta, Indonesia (1G)   | 641 Mbits/sec   | 40.5 Mbits/sec
 Clouvider       | NYC, NY, US (10G)         | 833 Mbits/sec   | 78.7 Mbits/sec
 Velocity Online | Tallahassee, FL, US (10G) | 766 Mbits/sec   | 138 Mbits/sec
 Clouvider       | Los Angeles, CA, US (10G) | 657 Mbits/sec   | 69.6 Mbits/sec   
 
 Geekbench 5 Benchmark Test:
 ---------------------------------
 Test            | Value
                 |
 Single Core     | 934
 Multi Core      | 18812
 Full Test       | https://browser.geekbench.com/v5/cpu/5039076
 
 https://github.com/masonr/yet-another-bench-script
 curl -sL yabs.sh | bash
 Server24 56 Cores with RAID5 NVMe, 256 GB RAM - 20201128
 
 Sat Nov 28 11:50:37 CET 2020
 
 Basic System Information:
 ---------------------------------
 Processor  : Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
 CPU cores  : 56 @ 1728.657 MHz
 AES-NI     : ✔ Enabled
 VM-x/AMD-V : ✔ Enabled
 RAM        : 251.5 GiB
 Swap       : 4.0 GiB
 Disk       : 191.9 GiB   
 
 fio Disk Speed Tests (Mixed R/W 50/50):
 ---------------------------------
 Block Size | 4k            (IOPS) | 64k           (IOPS)
   ------   | ---            ----  | ----           ----
 Read       | 309.13 MB/s  (77.2k) | 427.73 MB/s   (6.6k)
 Write      | 309.95 MB/s  (77.4k) | 429.99 MB/s   (6.7k)
 Total      | 619.08 MB/s (154.7k) | 857.73 MB/s  (13.4k)
            |                      |
 Block Size | 512k          (IOPS) | 1m            (IOPS)
   ------   | ---            ----  | ----           ----
 Read       | 390.30 MB/s    (762) | 506.25 MB/s    (494)
 Write      | 411.04 MB/s    (802) | 539.96 MB/s    (527)
 Total      | 801.35 MB/s   (1.5k) | 1.04 GB/s     (1.0k)

 iperf3 Network Speed Tests (IPv4):
 ---------------------------------
 Provider        | Location (Link)           | Send Speed      | Recv Speed
                 |                           |                 |
 Clouvider       | London, UK (10G)          | busy            | busy
 Online.net      | Paris, FR (10G)           | 874 Mbits/sec   | 505 Mbits/sec
 WorldStream     | The Netherlands (10G)     | 906 Mbits/sec   | 804 Mbits/sec
 Biznet          | Jakarta, Indonesia (1G)   | 691 Mbits/sec   | 42.5 Mbits/sec
 Clouvider       | NYC, NY, US (10G)         | 721 Mbits/sec   | 96.1 Mbits/sec
 Velocity Online | Tallahassee, FL, US (10G) | 722 Mbits/sec   | 180 Mbits/sec
 Clouvider       | Los Angeles, CA, US (10G) | 573 Mbits/sec   | 63.1 Mbits/sec 
 
 Geekbench 5 Benchmark Test:
 ---------------------------------
 Test            | Value
                 |
 Single Core     | 929
 Multi Core      | 18811
 Full Test       | https://browser.geekbench.com/v5/cpu/5039758

 Server24 56 Cores configured with RAID0 4x Intel Optane NVMe, 256 GB RAM - 20201128
 
 Basic System Information:
 ---------------------------------
 Processor  : Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
 CPU cores  : 56 @ 2096.213 MHz
 AES-NI     : ✔ Enabled
 VM-x/AMD-V : ✔ Enabled
 RAM        : 251.5 GiB
 Swap       : 4.0 GiB
 Disk       : 218.7 GiB
 
 Preparing system for disk tests...curl -s -4 --connect-timeout 5 --retry 5 --retry-delay 0 https://raw.githubusercontent.com/masonr/yet-another-bench-script/master/bin/fio_x64 -o ./2020-11-28T12_06_42+01_00/disk/fio
 fio Disk Speed Tests (Mixed R/W 50/50):
 ---------------------------------
 Block Size | 4k            (IOPS) | 64k           (IOPS)
   ------   | ---            ----  | ----           ----
 Read       | 768.58 MB/s (192.1k) | 1.11 GB/s    (17.4k)
 Write      | 770.61 MB/s (192.6k) | 1.12 GB/s    (17.5k)
 Total      | 1.53 GB/s   (384.7k) | 2.23 GB/s    (34.9k)
            |                      |
 Block Size | 512k          (IOPS) | 1m            (IOPS)
   ------   | ---            ----  | ----           ----
 Read       | 1.08 GB/s     (2.1k) | 1.07 GB/s     (1.0k)
 Write      | 1.13 GB/s     (2.2k) | 1.14 GB/s     (1.1k)
 Total      | 2.22 GB/s     (4.3k) | 2.21 GB/s     (2.1k)
 
 iperf3 Network Speed Tests (IPv4):
 ---------------------------------
 Provider        | Location (Link)           | Send Speed      | Recv Speed
                 |                           |                 |
 Clouvider       | London, UK (10G)          | 875 Mbits/sec   | 640 Mbits/sec
 Online.net      | Paris, FR (10G)           | 886 Mbits/sec   | 549 Mbits/sec
 WorldStream     | The Netherlands (10G)     | 910 Mbits/sec   | 799 Mbits/sec
 Biznet          | Jakarta, Indonesia (1G)   | 651 Mbits/sec   | 42.0 Mbits/sec
 Clouvider       | NYC, NY, US (10G)         | 793 Mbits/sec   | 103 Mbits/sec
 Velocity Online | Tallahassee, FL, US (10G) | 728 Mbits/sec   | 176 Mbits/sec
 Clouvider       | Los Angeles, CA, US (10G) | 450 Mbits/sec   | 56.0 Mbits/sec 

 Geekbench 5 Benchmark Test:
 ---------------------------------
 Test            | Value
                 |
 Single Core     | 926
 Multi Core      | 18879
 Full Test       | https://browser.geekbench.com/v5/cpu/5040005

So we know for sure that RAID0 rules, but the sheer outperformance of RAID10 vs RAID5 clearly shows that with NVMe disks, the CPU becomes the limiting factor. And as my colleague Markus pointed out, probably this is the reason we see no RAID5 or RAID6 controllers on the market for NVMe disks – the sheer computing power of a CPU is barely enough to cope with SSDs.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.