DELL R710服务器RAID的IOPS测试

最近有个想法就是对公司机房不同型号的机器测试下他们在不同级别的RAID下IOPS的值，然后做个比较。目前刚好调拨了一台DELL R710机器，就先对他进行测试，其他机型后续有机会再测试。

一、服务器配置

品牌：DELL

型号：PowerEdge R710

CPU：2 * Intel(R) Xeon(R) CPU L5639 @ 2.13GHz

内存：12 * 8G

阵列卡：PERC 6/i

硬盘：1 * 600G 15k SAS + 4*2T 7.5k

系统：Centos 7

二、FIO配置

之前写过一篇文章linux使用FIO测试磁盘IO性能，具体的安装操作请参考上面的。

这边整理下fio命令输出的含义

read # 顺序读

write # 顺序写

rw,readwrite # 顺序混合读写

randwrite # 随机写

randread # 随机读

randrw # 随机混合读写

bw # 带宽，单位KB/s

iops # 每秒钟的IO数

runt # 总运行时间

lat (msec) # 延迟(毫秒)，msec为毫秒，usec为微秒

三、开始测试

1、4块盘做RAID 0

# fio -filename=/dev/sdb1 -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=100 -group_reporting -name=nolinux_test
nolinux_test: (g=0): rw=randrw, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, iodepth=1
...
fio-2.1.10
Starting 30 threads
Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [100.0% done] [10789KB/4427KB/0KB /s] [674/276/0 iops] [eta 00m:00s]
nolinux_test: (groupid=0, jobs=30): err= 0: pid=4286: Tue Dec 15 10:15:18 2015
read : io=902560KB, bw=9015.6KB/s, iops=563, runt=100117msec
clat (usec): min=185, max=1178.6K, avg=53052.13, stdev=73652.05
lat (usec): min=185, max=1178.6K, avg=53052.48, stdev=73652.05
clat percentiles (msec):
| 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 9], 20.00th=[ 12],
| 30.00th=[ 16], 40.00th=[ 21], 50.00th=[ 27], 60.00th=[ 38],
| 70.00th=[ 53], 80.00th=[ 78], 90.00th=[ 125], 95.00th=[ 180],
| 99.00th=[ 367], 99.50th=[ 474], 99.90th=[ 742], 99.95th=[ 816],
| 99.99th=[ 1012]
bw (KB /s): min= 13, max= 766, per=3.44%, avg=310.41, stdev=132.18
write: io=361040KB, bw=3606.2KB/s, iops=225, runt=100117msec
clat (usec): min=82, max=43986, avg=362.90, stdev=965.29
lat (usec): min=83, max=43987, avg=363.88, stdev=965.29
clat percentiles (usec):
| 1.00th=[ 92], 5.00th=[ 104], 10.00th=[ 131], 20.00th=[ 143],
| 30.00th=[ 151], 40.00th=[ 173], 50.00th=[ 193], 60.00th=[ 205],
| 70.00th=[ 227], 80.00th=[ 245], 90.00th=[ 540], 95.00th=[ 1496],
| 99.00th=[ 3088], 99.50th=[ 3792], 99.90th=[ 8640], 99.95th=[19072],
| 99.99th=[39168]
bw (KB /s): min= 12, max= 572, per=3.64%, avg=131.30, stdev=78.65
lat (usec) : 100=0.98%, 250=22.23%, 500=2.45%, 750=0.28%, 1000=0.46%
lat (msec) : 2=1.28%, 4=1.43%, 10=9.80%, 20=18.22%, 50=20.17%
lat (msec) : 100=12.54%, 250=8.43%, 500=1.43%, 750=0.23%, 1000=0.06%
lat (msec) : 2000=0.01%
cpu : usr=0.02%, sys=0.07%, ctx=79324, majf=0, minf=8
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=56410/w=22565/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
READ: io=902560KB, aggrb=9015KB/s, minb=9015KB/s, maxb=9015KB/s, mint=100117msec, maxt=100117msec
WRITE: io=361040KB, aggrb=3606KB/s, minb=3606KB/s, maxb=3606KB/s, mint=100117msec, maxt=100117msec

Disk stats (read/write):
sdb: ios=56356/40523, merge=0/145, ticks=2987510/35649, in_queue=3024500, util=100.00%

2、4块盘做RAID 5

# fio -filename=/dev/sdb1 -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=100 -group_reporting -name=nolinux_test
nolinux_test: (g=0): rw=randrw, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, iodepth=1
...
fio-2.1.10
Starting 30 threads
Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [100.0% done] [9504KB/3792KB/0KB /s] [594/237/0 iops] [eta 00m:00s]
nolinux_test: (groupid=0, jobs=30): err= 0: pid=2199: Tue Dec 15 10:28:26 2015
read : io=613232KB, bw=6127.4KB/s, iops=382, runt=100082msec
clat (usec): min=171, max=4002.1K, avg=76735.03, stdev=158366.22
lat (usec): min=172, max=4002.1K, avg=76735.41, stdev=158366.23
clat percentiles (msec):
| 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 14],
| 30.00th=[ 18], 40.00th=[ 24], 50.00th=[ 33], 60.00th=[ 45],
| 70.00th=[ 64], 80.00th=[ 96], 90.00th=[ 169], 95.00th=[ 277],
| 99.00th=[ 693], 99.50th=[ 1004], 99.90th=[ 1958], 99.95th=[ 2147],
| 99.99th=[ 3916]
bw (KB /s): min= 3, max= 759, per=3.88%, avg=237.85, stdev=160.57
write: io=251312KB, bw=2511.7KB/s, iops=156, runt=100082msec
clat (usec): min=83, max=3870.9K, avg=3781.44, stdev=98297.49
lat (usec): min=84, max=3870.9K, avg=3782.42, stdev=98297.49
clat percentiles (usec):
| 1.00th=[ 95], 5.00th=[ 127], 10.00th=[ 141], 20.00th=[ 157],
| 30.00th=[ 185], 40.00th=[ 197], 50.00th=[ 211], 60.00th=[ 231],
| 70.00th=[ 253], 80.00th=[ 306], 90.00th=[ 1624], 95.00th=[ 2992],
| 99.00th=[29056], 99.50th=[41728], 99.90th=[75264], 99.95th=[3719168],
| 99.99th=[3883008]
bw (KB /s): min= 3, max= 572, per=4.69%, avg=117.70, stdev=81.72
lat (usec) : 100=0.54%, 250=19.57%, 500=4.36%, 750=0.39%, 1000=0.38%
lat (msec) : 2=1.46%, 4=1.80%, 10=8.04%, 20=16.66%, 50=20.72%
lat (msec) : 100=12.55%, 250=9.40%, 500=2.80%, 750=0.70%, 1000=0.26%
lat (msec) : 2000=0.29%, >=2000=0.09%
cpu : usr=0.02%, sys=0.05%, ctx=54331, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=38327/w=15707/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
READ: io=613232KB, aggrb=6127KB/s, minb=6127KB/s, maxb=6127KB/s, mint=100082msec, maxt=100082msec
WRITE: io=251312KB, aggrb=2511KB/s, minb=2511KB/s, maxb=2511KB/s, mint=100082msec, maxt=100082msec

Disk stats (read/write):
sdb: ios=38279/30678, merge=3/136, ticks=2936846/123280, in_queue=3061101, util=100.00%

3、4块盘做RAID 6

# fio -filename=/dev/sdb1 -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=100 -group_reporting -name=nolinux_test
nolinux_test: (g=0): rw=randrw, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, iodepth=1
...
fio-2.1.10
Starting 30 threads
Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [100.0% done] [9200KB/4592KB/0KB /s] [575/287/0 iops] [eta 00m:00s]
nolinux_test: (groupid=0, jobs=30): err= 0: pid=2198: Tue Dec 15 10:43:00 2015
read : io=561360KB, bw=5608.4KB/s, iops=350, runt=100093msec
clat (usec): min=76, max=4208.3K, avg=83711.78, stdev=178042.39
lat (usec): min=76, max=4208.3K, avg=83712.11, stdev=178042.40
clat percentiles (msec):
| 1.00th=[ 6], 5.00th=[ 9], 10.00th=[ 11], 20.00th=[ 16],
| 30.00th=[ 22], 40.00th=[ 30], 50.00th=[ 40], 60.00th=[ 52],
| 70.00th=[ 71], 80.00th=[ 100], 90.00th=[ 172], 95.00th=[ 277],
| 99.00th=[ 807], 99.50th=[ 1139], 99.90th=[ 1975], 99.95th=[ 3982],
| 99.99th=[ 4146]
bw (KB /s): min= 3, max= 954, per=3.92%, avg=220.00, stdev=139.01
write: io=229424KB, bw=2292.2KB/s, iops=143, runt=100093msec
clat (usec): min=72, max=4101.1K, avg=4457.37, stdev=92933.19
lat (usec): min=72, max=4101.1K, avg=4458.29, stdev=92933.19
clat percentiles (usec):
| 1.00th=[ 90], 5.00th=[ 100], 10.00th=[ 125], 20.00th=[ 141],
| 30.00th=[ 149], 40.00th=[ 167], 50.00th=[ 197], 60.00th=[ 229],
| 70.00th=[ 262], 80.00th=[ 1112], 90.00th=[ 4448], 95.00th=[ 7072],
| 99.00th=[59136], 99.50th=[76288], 99.90th=[107008], 99.95th=[2539520],
| 99.99th=[4112384]
bw (KB /s): min= 3, max= 458, per=4.68%, avg=107.20, stdev=74.68
lat (usec) : 100=1.44%, 250=18.39%, 500=2.65%, 750=0.39%, 1000=0.26%
lat (msec) : 2=0.98%, 4=2.00%, 10=7.89%, 20=14.30%, 50=21.84%
lat (msec) : 100=15.61%, 250=10.17%, 500=2.58%, 750=0.69%, 1000=0.36%
lat (msec) : 2000=0.39%, >=2000=0.08%
cpu : usr=0.01%, sys=0.04%, ctx=49650, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=35085/w=14339/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
READ: io=561360KB, aggrb=5608KB/s, minb=5608KB/s, maxb=5608KB/s, mint=100093msec, maxt=100093msec
WRITE: io=229424KB, aggrb=2292KB/s, minb=2292KB/s, maxb=2292KB/s, mint=100093msec, maxt=100093msec

Disk stats (read/write):
sdb: ios=35025/27105, merge=0/72, ticks=2931793/173511, in_queue=3106873, util=99.99%

4、4块盘做RAID 10

# fio -filename=/dev/sdb1 -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=100 -group_reporting -name=nolinux_test
nolinux_test: (g=0): rw=randrw, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, iodepth=1
...
fio-2.1.10
Starting 30 threads
Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [100.0% done] [6656KB/2496KB/0KB /s] [416/156/0 iops] [eta 00m:00s]
nolinux_test: (groupid=0, jobs=30): err= 0: pid=2220: Tue Dec 15 11:02:45 2015
read : io=657040KB, bw=6565.4KB/s, iops=410, runt=100077msec
clat (usec): min=92, max=2180.4K, avg=72878.19, stdev=164493.62
lat (usec): min=92, max=2180.4K, avg=72878.53, stdev=164493.62
clat percentiles (msec):
| 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 11], 20.00th=[ 15],
| 30.00th=[ 20], 40.00th=[ 27], 50.00th=[ 36], 60.00th=[ 47],
| 70.00th=[ 60], 80.00th=[ 81], 90.00th=[ 129], 95.00th=[ 208],
| 99.00th=[ 898], 99.50th=[ 1303], 99.90th=[ 2024], 99.95th=[ 2073],
| 99.99th=[ 2114]
bw (KB /s): min= 7, max= 665, per=3.92%, avg=257.58, stdev=132.65
write: io=267312KB, bw=2671.7KB/s, iops=166, runt=100077msec
clat (usec): min=82, max=44946, avg=452.60, stdev=1261.13
lat (usec): min=82, max=44947, avg=453.54, stdev=1261.14
clat percentiles (usec):
| 1.00th=[ 94], 5.00th=[ 104], 10.00th=[ 135], 20.00th=[ 151],
| 30.00th=[ 157], 40.00th=[ 181], 50.00th=[ 201], 60.00th=[ 225],
| 70.00th=[ 245], 80.00th=[ 274], 90.00th=[ 724], 95.00th=[ 2256],
| 99.00th=[ 3888], 99.50th=[ 4832], 99.90th=[18816], 99.95th=[27520],
| 99.99th=[43776]
bw (KB /s): min= 6, max= 480, per=4.33%, avg=115.68, stdev=75.82
lat (usec) : 100=0.73%, 250=20.55%, 500=4.53%, 750=0.26%, 1000=0.20%
lat (msec) : 2=0.97%, 4=1.84%, 10=6.65%, 20=14.97%, 50=22.88%
lat (msec) : 100=16.28%, 250=7.29%, 500=1.47%, 750=0.52%, 1000=0.27%
lat (msec) : 2000=0.43%, >=2000=0.15%
cpu : usr=0.01%, sys=0.05%, ctx=58127, majf=0, minf=8
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=41065/w=16707/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
READ: io=657040KB, aggrb=6565KB/s, minb=6565KB/s, maxb=6565KB/s, mint=100077msec, maxt=100077msec
WRITE: io=267312KB, aggrb=2671KB/s, minb=2671KB/s, maxb=2671KB/s, mint=100077msec, maxt=100077msec

Disk stats (read/write):
sdb: ios=41009/32303, merge=3/85, ticks=2988227/50660, in_queue=3040152, util=100.00%

四、结果总结

从结果看出，RAID 5因其在成本，速度，冗余三方面保持了平衡，所以被我们广泛应用；如果对空间需求不高的话，RAID 10也是不错的选择。