A Random Walk Through Idea Space: FusionIO ioDrive

Well, I got my hands on one of the fusion-io ioDrives a couple of weeks ago. unfortunately they do not work in the version of VMware ESX that we are using, though they are working on drivers for the 64bit ESX4. I did not have time to set up a physical machine to test on our application running SQL server 2005 so I have just quickly done some IO benchmarks in Linux at home. I was going to test btrfs and its SSD mode at the same time it but hit too many problems trying to get the drivers and the btrfs module in the kernel together.

First setup a 4 drive raid0 array for comparison

root@George:/home/tom# fdisk -l | grep 500
Disk /dev/sdb doesn't contain a valid partition table
Disk /dev/sdc doesn't contain a valid partition table
Disk /dev/sdd doesn't contain a valid partition table
Disk /dev/sde doesn't contain a valid partition table
Disk /dev/sdb: 500.1 GB, 500107862016 bytes
Disk /dev/sdc: 500.1 GB, 500107862016 bytes
Disk /dev/sdd: 500.1 GB, 500107862016 bytes
Disk /dev/sde: 500.1 GB, 500107862016 bytes

root@George:/home/tom# mdadm --create /dev/md0 -l 0 -n 4 /dev/sd[bcde]
mdadm: array /dev/md0 started.
root@George:/home/tom# mkfs.ext2 /dev/md0
root@George:/home/tom# mkfs.ext2 /dev/fioa

fio

I was looking for an iometer-a-like for linux to quickly get some semi-meaningful results (bonnie++ was returning results saying it was too quick to measure or something) Fio lets you create a text file description of a workload, with a choice of IO libs and loads of options, you can set concurrent jobs also, see this excellent linux.com article for more info

random-read-test-aio-32thread-20G.fio

[random-read]
rw=randread
size=20G
ioengine=libaio
iodepth=32
direct=1
invalidate=1

root@George:/fusionio# fio /fio/random-read-test-aio-32thread-20G.fio
random-read: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
Starting 1 process
random-read: Laying out IO file(s) (1 file(s) / 20480MiB)
Jobs: 1 (f=1): [r] [100.0% done] [139M/0K /s] [35K/0 iops] [eta 00m:00s]
random-read: (groupid=0, jobs=1): err= 0: pid=25705
  read : io=20480MiB, bw=157614KiB/s, iops=39403, runt=133056msec
    slat (usec): min=4096, max=4096, avg=4096.00, stdev= 0.00
    clat (usec): min=324, max=325739, avg=788.05, stdev=1238.41
    bw (KiB/s) : min=45800, max=196536, per=100.24%, avg=157996.42, stdev=22439.38
  cpu          : usr=9.23%, sys=71.81%, ctx=2180058, majf=1, minf=698
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued r/w: total=5242880/0, short=0/0
     lat (usec): 500=0.01%, 750=78.27%, 1000=10.86%
     lat (msec): 2=9.31%, 4=1.11%, 10=0.30%, 20=0.10%, 50=0.04%
     lat (msec): 100=0.01%, 500=0.01%
Run status group 0 (all jobs):
READ: io=20480MiB, aggrb=157614KiB/s, minb=157614KiB/s, maxb=157614KiB/s, mint=133056msec, maxt=133056msec


Disk stats (read/write):
  fioa: ios=5244775/2, merge=0/0, ticks=303464/0, in_queue=0, util=0.00%

random-write-test-aio-32thread-20G.fio

[random-write]
rw=randwrite
size=20G
ioengine=libaio
iodepth=32
direct=1
invalidate=1

root@George:/fusionio# fio /fio/random-write-test-aio-32thread-20G.fio
random-write: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
Starting 1 process
random-write: Laying out IO file(s) (1 file(s) / 20480MiB)
Jobs: 1 (f=1): [w] [100.0% done] [0K/18210K /s] [0/4446 iops] [eta 00m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=7105
  write: io=20480MiB, bw=23406KiB/s, iops=5851, runt=895978msec
    slat (usec): min=4096, max=4096, avg=4096.00, stdev= 0.00
    clat (msec): min=1, max=322, avg= 5.30, stdev= 6.29
    bw (KiB/s) : min=    0, max=94080, per=99.99%, avg=23404.82, stdev=13749.60
  cpu          : usr=3.19%, sys=27.08%, ctx=5303118, majf=0, minf=4369
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/5242880, short=0/0
     lat (msec): 2=14.96%, 4=46.44%, 10=17.79%, 20=20.45%, 50=0.23%
     lat (msec): 100=0.06%, 250=0.06%, 500=0.01%
Run status group 0 (all jobs):
  WRITE: io=20480MiB, aggrb=23406KiB/s, minb=23406KiB/s, maxb=23406KiB/s, mint=895978msec, maxt=895978msec
Disk stats (read/write):
  fioa: ios=164/5339709, merge=0/0, ticks=28/961368, in_queue=0, util=0.00%

Now the RAID0 array,took all night to complete the same tests.

root@George:/raid0# fio /fio/random-read-test-aio-32thread-20G.fio ;
random-read: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [4136K/0K /s] [1010/0 iops] [eta 00m:00s]
random-read: (groupid=0, jobs=1): err= 0: pid=20079
  read : io=20480MiB, bw=3475KiB/s, iops=868, runt=6033707msec
    slat (usec): min=4096, max=4096, avg=4096.00, stdev= 0.00
    clat (usec): min=3, max=1048K, avg=36802.65, stdev=37338.47
    bw (KiB/s) : min=  982, max= 4367, per=100.10%, avg=3478.65, stdev=279.09
  cpu          : usr=0.55%, sys=1.84%, ctx=4607245, majf=0, minf=30634
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued r/w: total=5242880/0, short=0/0
     lat (usec): 4=0.01%, 50=0.01%, 100=0.01%, 250=0.03%, 500=0.09%
     lat (usec): 750=0.01%, 1000=0.01%
     lat (msec): 2=0.02%, 4=0.93%, 10=16.14%, 20=26.65%, 50=32.57%
     lat (msec): 100=16.79%, 250=6.56%, 500=0.19%, 750=0.01%, 1000=0.01%
     lat (msec): 2000=0.01%
Run status group 0 (all jobs):
   READ: io=20480MiB, aggrb=3475KiB/s, minb=3475KiB/s, maxb=3475KiB/s, mint=6033707msec, maxt=6033707msec
Disk stats (read/write):
  md0: ios=5242880/4900, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    sdb: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    sdc: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    sdd: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    sde: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%


root@George:/raid0# fio /fio/random-write-test-aio-32thread-20G.fio

random-write: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=32
Starting 1 process
random-write: Laying out IO file(s) (1 file(s) / 20480MiB)
Jobs: 1 (f=1): [w] [100.0% done] [0K/9427K /s] [0/2301 iops] [eta 00m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=2789
  write: io=20480MiB, bw=10075KiB/s, iops=2518, runt=2081336msec
    slat (usec): min=4096, max=4096, avg=4096.00, stdev= 0.00
    clat (msec): min=1, max=773, avg=12.31, stdev= 7.27
    bw (KiB/s) : min=    0, max=20552, per=100.09%, avg=10083.67, stdev=2143.10
  cpu          : usr=1.62%, sys=11.61%, ctx=5332417, majf=0, minf=10120
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued r/w: total=0/5242880, short=0/0
     lat (msec): 2=0.01%, 4=0.01%, 10=23.86%, 20=75.38%, 50=0.70%
     lat (msec): 100=0.04%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
Run status group 0 (all jobs):
  WRITE: io=20480MiB, aggrb=10075KiB/s, minb=10075KiB/s, maxb=10075KiB/s, mint=2081336msec, maxt=2081336msec
Disk stats (read/write):
  md0: ios=159/5353967, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    sdb: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    sdc: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    sdd: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%
    sde: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=nan%

Key figures are IOPS, bw (bandwith) and completion time.

IOPS: read - 39403/868 ~45x write - 5851/2518 ~2x

bw read - 157614/3475 ~45x write - 23406/10075 ~2x

Time: read - 133/6033 ~45x write - 895/2081 ~2x

I am surprised the array wrote so fast ~800 IOPS for read and write was my expectation, I did not have time to repeat with a different IO lib and cant explain why random writes would be 3x faster than random reads on a RAID0 array so I think its to do with libaio.

In short my home PC temporarily had read IOPS equivalent to 200 hard disks spinning at 15'000 RPM, ace.

Prices are as follows:

80GB ioDrive (SLC) US$3,600
160GB ioDrive (SLC) US$7,200
320GB ioDrive (MLC) US$14,400
320GB ioDrive Duo (SLC) US$11,900
640GB ioDrive Duo (MLC) US$9,795

See http://www.fusionio.com/Products.aspx for more information.

Menu

fio

random-read-test-aio-32thread-20G.fio

random-write-test-aio-32thread-20G.fio