Everything you ever wanted to know about RAID (Part 4 of 4)

By Alasdair Lumsden on 7 Nov 2013

Part 4: Performance Benchmarks

  1. An overview of RAID
  2. RAID Levels
  3. RAID Hardware & Software (including ZFS)
  4. Performance Benchmarks

Welcome to Part 4 of our series of blog posts on RAID – Redundant Array of Inexpensive Disks. In this post, we’ll be going through the performance benchmarks we’ve performed using both Hardware RAID and ZFS Software RAID.

Since our entire platform uses SAS Host Bus Adapters and ZFS RAID, rather than using Hardware RAID cards, we’ve split the tests into two sections. The first section covers a fair comparison between HW RAID 5 and 10 plus ZFS RAID 5 and 10 on a Dell Perc H310 HW RAID card. The second tests covers a range of ZFS RAID tests on an LSI 9207-8i SAS HBA, and includes some additional ZFS RAID types (such as RAIDZ2 (RAID6) and Hybrid SSD accelerated pools).

Test Hardware

For the test we’re using a standard Dell R620 server of which we have many. This has the following specifications

Model: Dell R620
CPUs: 2 x Intel E5-2650 CPUs @ 2.0GHz (8 cores per CPU)
RAM: 192GB (24x8GB)
Disks 6 x 300GB 15k SAS Drives (Seagate ST9300653SS)

Test Methodology

For the tests we’re running SmartOS 2013-10-11, and running vdbench, which is a UNIX based disk performance tool. Our vdbench test script can be downloaded here. We run 3 tests, 100% reads (with ZFS read caching disabled, otherwise the dataset is cached in RAM and meaningless), 100% async writes, and 100% sync writes. This should give a fairly representative test of the relative disk performance.

The raw numbers can also be found on our Google Docs Spreadsheet.

HW & ZFS RAID 5 vs RAID 10

PERC H310

These first set of tests are conducted on a Dell Perc H310 Controller. This is a common budget Dell controller, with no battery backed cache. I would have liked to have seen the performance improvement with a battery backed cache, but since we use ZFS, we don’t have any such cards available.

The results are as follows:

hw-zfs-read-new4

hw-zfs-write-async-new4

hw-zfs-write-sync-new2

What can we conclude from the above? Well we can see some interesting results. For Hardware RAID, at least with a Dell H310, RAID10 is faster across the board, especially for writes where it’s between 2 to 5 times faster depend on async vs sync. This conforms to conventional wisdom on the topic and is what I’d expect.

For ZFS, RAID10 is much faster for reads, but surprisingly considerably slower for writes. This is an interesting result. ZFS is a very non-traditional filesystem, so benchmarking your workload becomes all the more important.

In terms of HW RAID vs ZFS RAID, it seems performance very much depends on your workload – whether it’s read heavy or write heavy, and whether your writes are synchronous or asynchronous. Although there’s no overall winner, ZFS RAID5 definitely has the edge over HW RAID5, whilst HW RAID10 beats ZFS RAID10.

ZFS RAID vs ZFS RAID

hba-9207-8i-large

I thought it would be worth conducting a series of benchmarks of the different ZFS configurations available, on the LSI 9207-8i SAS HBA cards that we use at EveryCity, to see the different performance characteristics we could get.

ZFS is unique in that it also supports "Hybrid Storage Pools", where it can make use of SSD drives as write caches (The ZIL or ZFS Intent Log, aka the SLOG) or read caches (The L2ARC or Level 2 Adaptive Replacement Cache). Using SSDs as caches can dramatically improve the overall performance of a storage array whilst still using cheaper larger spinning disks. Indeed, we use SSDs to accelerate our ZFS NFS storage product (8GB STEC ZeusRAM for the write cache and 512GB 2.5″ SATA SSDs for the read cache), and we used Sun F20 PCIe SSD cards in our iSCSI SAN storage product.

For our cloud platform, we’re presently using either all-SAS drives or all-SSDs, and I was curious to see how the addition of a single SSD device would change the performance landmark.

zfs-read

zfs-write-async

zfs-write-sync

So, we have some very interesting results. The first thing to note is that the LSI 9207-8i card has given considerably better results overall for the ZFS RAID. I would expect this, as the 9207-8i is a pure SAS HBA with minimal RAID capabilities, designed for providing direct access to the drives from the operating system, which is exactly what ZFS wants. Indeed, ZFS RAID10 now beats the HW RAID10 for both reads and async writes (but not sync writes).

This may stem from any caching or ordering behaviour the Dell PERC H310 cards exhibit. To expose the individual drives through to the operating system, I had to create 6 individual RAID0 volumes on the card. Clearly with ZFS, using a pure SAS HBA is of benefit.

Next, we can still see that ZFS RAID10 beats RAIDZ(5) and RAIDZ2(6) for reads, and RAIDZ and RAIDZ2 beat RAID10 for writes.

Finally, we can see that the SSD write cache (ZIL/SLOG) device improves synchronous write performance substantially. However, the write cache device slightly slows down async writes for RAIDZ, whilst it slightly speeds them up for RAIDZ2. This may be within the margin of error, but it’s hard to say. When pushing a system to the limits, small differences may become pronounced.

Conclusions

In conclusion, as with everything, what to use depends entirely on your workload, hardware, and budget. The biggest takeaway I’ve had from this benchmarking experience has been to discover that ZFS RAIDZ and RAIDZ2 write performance beats ZFS RAID10 write performance. Since our systems are write heavy (thanks to the efficient ZFS ARC cache), RAIDZ or RAIDZ2 is the obvious choice – yet we’ve been using RAID10. So here, we can easily improve both our capacity and performance, thanks to these benchmarks.

On the other hand, for hardware RAID, at least for our setup, conventional wisdom still holds – RAID10 beats RAID5 for both read and write performance.

If you’re interested in more information about anything in this series of blog posts, please don’t hesitate to get in touch with me via the comments section below. Thanks for reading!