Today we ran some tests and benchmarks and while in the past with mechanical hard drives the hit from the overhead of running checksums etc for RAID5 and/or RAID6 was negligible, with SSD it became a factor and with the latest generation Optane disks the difference is just huge. Compare the following numbers taken on a 56 vCore server with 4x Intel Optane NVMe drives configured as RAID10, RAID5 and RAID0:
Server24 56 Cores with RAID10 NVMe, 256 GB RAM - 20201128 Sat Nov 28 11:04:33 CET 2020 Basic System Information: --------------------------------- Processor : Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz CPU cores : 56 @ 1707.405 MHz AES-NI : ✔ Enabled VM-x/AMD-V : ✔ Enabled RAM : 251.5 GiB Swap : 4.0 GiB Disk : 165.3 GiB fio Disk Speed Tests (Mixed R/W 50/50): --------------------------------- Block Size | 4k (IOPS) | 64k (IOPS) ------ | --- ---- | ---- ---- Read | 527.42 MB/s (131.8k) | 615.91 MB/s (9.6k) Write | 528.81 MB/s (132.2k) | 619.15 MB/s (9.6k) Total | 1.05 GB/s (264.0k) | 1.23 GB/s (19.2k) | | Block Size | 512k (IOPS) | 1m (IOPS) ------ | --- ---- | ---- ---- Read | 565.11 MB/s (1.1k) | 597.10 MB/s (583) Write | 595.13 MB/s (1.1k) | 636.87 MB/s (621) Total | 1.16 GB/s (2.2k) | 1.23 GB/s (1.2k) iperf3 Network Speed Tests (IPv4): --------------------------------- Provider | Location (Link) | Send Speed | Recv Speed | | | Clouvider | London, UK (10G) | 888 Mbits/sec | 650 Mbits/sec Online.net | Paris, FR (10G) | 880 Mbits/sec | 533 Mbits/sec WorldStream | The Netherlands (10G) | 913 Mbits/sec | 815 Mbits/sec Biznet | Jakarta, Indonesia (1G) | 641 Mbits/sec | 40.5 Mbits/sec Clouvider | NYC, NY, US (10G) | 833 Mbits/sec | 78.7 Mbits/sec Velocity Online | Tallahassee, FL, US (10G) | 766 Mbits/sec | 138 Mbits/sec Clouvider | Los Angeles, CA, US (10G) | 657 Mbits/sec | 69.6 Mbits/sec Geekbench 5 Benchmark Test: --------------------------------- Test | Value | Single Core | 934 Multi Core | 18812 Full Test | https://browser.geekbench.com/v5/cpu/5039076 https://github.com/masonr/yet-another-bench-script curl -sL yabs.sh | bash Server24 56 Cores with RAID5 NVMe, 256 GB RAM - 20201128 Sat Nov 28 11:50:37 CET 2020 Basic System Information: --------------------------------- Processor : Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz CPU cores : 56 @ 1728.657 MHz AES-NI : ✔ Enabled VM-x/AMD-V : ✔ Enabled RAM : 251.5 GiB Swap : 4.0 GiB Disk : 191.9 GiB fio Disk Speed Tests (Mixed R/W 50/50): --------------------------------- Block Size | 4k (IOPS) | 64k (IOPS) ------ | --- ---- | ---- ---- Read | 309.13 MB/s (77.2k) | 427.73 MB/s (6.6k) Write | 309.95 MB/s (77.4k) | 429.99 MB/s (6.7k) Total | 619.08 MB/s (154.7k) | 857.73 MB/s (13.4k) | | Block Size | 512k (IOPS) | 1m (IOPS) ------ | --- ---- | ---- ---- Read | 390.30 MB/s (762) | 506.25 MB/s (494) Write | 411.04 MB/s (802) | 539.96 MB/s (527) Total | 801.35 MB/s (1.5k) | 1.04 GB/s (1.0k) iperf3 Network Speed Tests (IPv4): --------------------------------- Provider | Location (Link) | Send Speed | Recv Speed | | | Clouvider | London, UK (10G) | busy | busy Online.net | Paris, FR (10G) | 874 Mbits/sec | 505 Mbits/sec WorldStream | The Netherlands (10G) | 906 Mbits/sec | 804 Mbits/sec Biznet | Jakarta, Indonesia (1G) | 691 Mbits/sec | 42.5 Mbits/sec Clouvider | NYC, NY, US (10G) | 721 Mbits/sec | 96.1 Mbits/sec Velocity Online | Tallahassee, FL, US (10G) | 722 Mbits/sec | 180 Mbits/sec Clouvider | Los Angeles, CA, US (10G) | 573 Mbits/sec | 63.1 Mbits/sec Geekbench 5 Benchmark Test: --------------------------------- Test | Value | Single Core | 929 Multi Core | 18811 Full Test | https://browser.geekbench.com/v5/cpu/5039758 Server24 56 Cores configured with RAID0 4x Intel Optane NVMe, 256 GB RAM - 20201128 Basic System Information: --------------------------------- Processor : Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz CPU cores : 56 @ 2096.213 MHz AES-NI : ✔ Enabled VM-x/AMD-V : ✔ Enabled RAM : 251.5 GiB Swap : 4.0 GiB Disk : 218.7 GiB Preparing system for disk tests...curl -s -4 --connect-timeout 5 --retry 5 --retry-delay 0 https://raw.githubusercontent.com/masonr/yet-another-bench-script/master/bin/fio_x64 -o ./2020-11-28T12_06_42+01_00/disk/fio fio Disk Speed Tests (Mixed R/W 50/50): --------------------------------- Block Size | 4k (IOPS) | 64k (IOPS) ------ | --- ---- | ---- ---- Read | 768.58 MB/s (192.1k) | 1.11 GB/s (17.4k) Write | 770.61 MB/s (192.6k) | 1.12 GB/s (17.5k) Total | 1.53 GB/s (384.7k) | 2.23 GB/s (34.9k) | | Block Size | 512k (IOPS) | 1m (IOPS) ------ | --- ---- | ---- ---- Read | 1.08 GB/s (2.1k) | 1.07 GB/s (1.0k) Write | 1.13 GB/s (2.2k) | 1.14 GB/s (1.1k) Total | 2.22 GB/s (4.3k) | 2.21 GB/s (2.1k) iperf3 Network Speed Tests (IPv4): --------------------------------- Provider | Location (Link) | Send Speed | Recv Speed | | | Clouvider | London, UK (10G) | 875 Mbits/sec | 640 Mbits/sec Online.net | Paris, FR (10G) | 886 Mbits/sec | 549 Mbits/sec WorldStream | The Netherlands (10G) | 910 Mbits/sec | 799 Mbits/sec Biznet | Jakarta, Indonesia (1G) | 651 Mbits/sec | 42.0 Mbits/sec Clouvider | NYC, NY, US (10G) | 793 Mbits/sec | 103 Mbits/sec Velocity Online | Tallahassee, FL, US (10G) | 728 Mbits/sec | 176 Mbits/sec Clouvider | Los Angeles, CA, US (10G) | 450 Mbits/sec | 56.0 Mbits/sec Geekbench 5 Benchmark Test: --------------------------------- Test | Value | Single Core | 926 Multi Core | 18879 Full Test | https://browser.geekbench.com/v5/cpu/5040005
So we know for sure that RAID0 rules, but the sheer outperformance of RAID10 vs RAID5 clearly shows that with NVMe disks, the CPU becomes the limiting factor. And as my colleague Markus pointed out, probably this is the reason we see no RAID5 or RAID6 controllers on the market for NVMe disks – the sheer computing power of a CPU is barely enough to cope with SSDs.