root@storage-ix:~# fio testdisk.ini
readtest: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=4
writetest: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=4
fio-2.16
Starting 2 processes
fio: file /dev/sdb exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/sdb exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
^Cbs: 2 (f=2): [r(1),w(1)] [0.0% done] [36KB/3744KB/0KB /s] [9/936/0 iops] [eta 39155d:04h:56m:18s]
fio: terminating on signal 2
Jobs: 1 (f=1): [r(1),_(1)] [0.0% done] [20KB/2160KB/0KB /s] [5/540/0 iops] [eta 39160d:20h:56m:11s]
readtest: (groupid=0, jobs=1): err= 0: pid=135177: Fri Apr 10 19:26:17 2020
read: io=196824KB, bw=49642B/s, iops=12, runt=4059954msec
slat (usec): min=5, max=95, avg=35.93, stdev= 4.01
clat (usec): min=535, max=5194.4K, avg=329927.55, stdev=506721.89
lat (usec): min=575, max=5194.4K, avg=329964.22, stdev=506721.95
clat percentiles (msec):
| 1.00th=[ 9], 5.00th=[ 15], 10.00th=[ 18], 20.00th=[ 25],
| 30.00th=[ 35], 40.00th=[ 56], 50.00th=[ 100], 60.00th=[ 184],
| 70.00th=[ 318], 80.00th=[ 537], 90.00th=[ 971], 95.00th=[ 1434],
| 99.00th=[ 2409], 99.50th=[ 2671], 99.90th=[ 3326], 99.95th=[ 3556],
| 99.99th=[ 4146]
lat (usec): 750=0.01%, 1000=0.01%
lat (msec): 2=0.06%, 4=0.15%, 10=1.39%, 20=11.86%, 50=24.66%
lat (msec): 100=11.86%, 250=15.75%, 500=12.99%, 750=7.19%, 1000=4.49%
lat (msec): 2000=7.45%, >=2000=2.14%
cpu: usr=0.03%, sys=0.05%, ctx=49453, majf=0, minf=13
IO depths: 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued: total=r=49206/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency: target=0, window=0, percentile=100.00%, depth=4
writetest: (groupid=0, jobs=1): err= 0: pid=135178: Fri Apr 10 19:26:17 2020
write: io=13514MB, bw=3409.3KB/s, iops=852, runt=4059188msec
slat (usec): min=3, max=182, avg=15.05, stdev=10.34
clat (usec): min=35, max=1216.1K, avg=4673.97, stdev=4252.08
lat (usec): min=44, max=1216.2K, avg=4689.37, stdev=4251.51
clat percentiles (usec):
| 1.00th=[ 326], 5.00th=[ 1096], 10.00th=[ 1608], 20.00th=[ 2320],
| 30.00th=[ 2896], 40.00th=[ 3472], 50.00th=[ 4080], 60.00th=[ 4704],
| 70.00th=[ 5536], 80.00th=[ 6560], 90.00th=[ 8256], 95.00th=[10176],
| 99.00th=[14784], 99.50th=[17024], 99.90th=[24448], 99.95th=[29056],
| 99.99th=[52992]
lat (usec): 50=0.03%, 100=0.10%, 250=0.66%, 500=0.81%, 750=1.05%
lat (usec): 1000=1.60%
lat (msec): 2=11.03%, 4=33.58%, 10=45.87%, 20=5.03%, 50=0.23%
lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec): 2000=0.01%
cpu: usr=0.59%, sys=1.58%, ctx=976798, majf=0, minf=9
IO depths: 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued: total=r=0/w=3459470/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency: target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
READ: io=196824KB, aggrb=48KB/s, minb=48KB/s, maxb=48KB/s, mint=4059954msec, maxt=4059954msec
WRITE: io=13514MB, aggrb=3409KB/s, minb=3409KB/s, maxb=3409KB/s, mint=4059188msec, maxt=4059188msec

Disk stats (read/write):
sdb: ios=49254/3459470, merge=0/0, ticks=16225180/15962664, in_queue=32188164, util=99.93%

Read — Enabled
Write — Instant Write-Back

root@storage-ix:~# fio testdisk.ini
readtest: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
writetest: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.16
Starting 2 processes
fio: file /dev/sdb exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/sdb exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
^Cbs: 2 (f=2): [r(1),w(1)] [0.0% done] [56KB/4464KB/0KB /s] [14/1116/0 iops] [eta 19578d:03h:55m:15s] 49s]
fio: terminating on signal 2

readtest: (groupid=0, jobs=1): err= 0: pid=119128: Fri Apr 10 14:53:46 2020
read: io=91952KB, bw=99458B/s, iops=24, runt=946717msec
slat (usec): min=3, max=143, avg=21.64, stdev= 4.06
clat (usec): min=276, max=6986.8K, avg=1316779.75, stdev=963039.96
lat (usec): min=291, max=6986.8K, avg=1316802.00, stdev=963040.24
clat percentiles (usec):
| 1.00th=[ 438], 5.00th=[55552], 10.00th=[242688], 20.00th=[411648],
| 30.00th=[618496], 40.00th=[856064], 50.00th=[1122304], 60.00th=[1433600],
| 70.00th=[1794048], 80.00th=[2179072], 90.00th=[2703360], 95.00th=[3096576],
| 99.00th=[3817472], 99.50th=[4112384], 99.90th=[4751360], 99.95th=[5210112],
| 99.99th=[5734400]
lat (usec): 500=1.58%, 750=0.22%, 1000=0.08%
lat (msec): 2=0.02%, 4=0.02%, 10=0.03%, 20=0.47%, 50=2.34%
lat (msec): 100=1.12%, 250=4.50%, 500=13.97%, 750=11.24%, 1000=9.90%
lat (msec): 2000=30.02%, >=2000=24.47%
cpu: usr=0.12%, sys=0.09%, ctx=22427, majf=0, minf=42
IO depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued: total=r=22988/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency: target=0, window=0, percentile=100.00%, depth=32
writetest: (groupid=0, jobs=1): err= 0: pid=119129: Fri Apr 10 14:53:46 2020
write: io=3325.8MB, bw=3598.6KB/s, iops=899, runt=946370msec
slat (usec): min=2, max=254, avg= 9.38, stdev= 6.30
clat (usec): min=99, max=2166.7K, avg=35533.69, stdev=84533.56
lat (usec): min=119, max=2166.7K, avg=35543.31, stdev=84534.42
clat percentiles (usec):
| 1.00th=[ 478], 5.00th=[ 628], 10.00th=[ 836], 20.00th=[ 1256],
| 30.00th=[ 1704], 40.00th=[ 2160], 50.00th=[ 2672], 60.00th=[ 9024],
| 70.00th=[20608], 80.00th=[34048], 90.00th=[89600], 95.00th=[228352],
| 99.00th=[374784], 99.50th=[440320], 99.90th=[675840], 99.95th=[872448],
| 99.99th=[1712128]
lat (usec): 100=0.01%, 250=0.09%, 500=1.71%, 750=6.19%, 1000=5.82%
lat (msec): 2=22.60%, 4=20.21%, 10=4.21%, 20=8.57%, 50=17.35%
lat (msec): 100=3.31%, 250=5.89%, 500=3.76%, 750=0.21%, 1000=0.04%
lat (msec): 2000=0.03%, >=2000=0.01%
cpu: usr=0.60%, sys=1.40%, ctx=604154, majf=0, minf=9
IO depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued: total=r=0/w=851386/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency: target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
READ: io=91952KB, aggrb=97KB/s, minb=97KB/s, maxb=97KB/s, mint=946717msec, maxt=946717msec
WRITE: io=3325.8MB, aggrb=3598KB/s, minb=3598KB/s, maxb=3598KB/s, mint=946370msec, maxt=946370msec

Disk stats (read/write):
sdb: ios=23033/851386, merge=0/0, ticks=30243240/30212112, in_queue=60458144, util=99.70%


readtest: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=4
writetest: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=4
fio-2.16
Starting 2 processes
fio: file /dev/sdb exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
fio: file /dev/sdb exceeds 32-bit tausworthe random generator.
fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning.
^Cbs: 2 (f=2): [r(1),w(1)] [0.0% done] [44KB/3128KB/0KB /s] [11/782/0 iops] [eta 36987d:14h:32m:35s] :50s]
fio: terminating on signal 2
Jobs: 1 (f=0): [f(1),_(1)] [100.0% done] [36KB/1008KB/0KB /s] [9/252/0 iops] [eta 00m:00s]
readtest: (groupid=0, jobs=1): err= 0: pid=129208: Fri Apr 10 17:40:29 2020
read: io=134300KB, bw=52552B/s, iops=12, runt=2616875msec
slat (usec): min=2, max=84, avg=35.09, stdev= 6.52
clat (usec): min=95, max=4144.6K, avg=311629.79, stdev=494456.53
lat (usec): min=112, max=4144.7K, avg=311665.57, stdev=494456.95
clat percentiles (usec):
| 1.00th=[ 143], 5.00th=[ 5728], 10.00th=[10432], 20.00th=[16320],
| 30.00th=[23936], 40.00th=[43264], 50.00th=[84480], 60.00th=[162816],
| 70.00th=[296960], 80.00th=[509952], 90.00th=[946176], 95.00th=[1384448],
| 99.00th=[2310144], 99.50th=[2637824], 99.90th=[3358720], 99.95th=[3653632],
| 99.99th=[4079616]
lat (usec): 100=0.01%, 250=3.80%, 500=0.51%, 750=0.01%, 1000=0.01%
lat (msec): 2=0.01%, 4=0.14%, 10=4.86%, 20=16.44%, 50=16.42%
lat (msec): 100=10.27%, 250=14.55%, 500=12.57%, 750=6.70%, 1000=4.48%
lat (msec): 2000=7.39%, >=2000=1.83%
cpu: usr=0.04%, sys=0.06%, ctx=33476, majf=0, minf=14
IO depths: 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued: total=r=33575/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency: target=0, window=0, percentile=100.00%, depth=4
writetest: (groupid=0, jobs=1): err= 0: pid=129209: Fri Apr 10 17:40:29 2020
write: io=8751.7MB, bw=3425.4KB/s, iops=856, runt=2616298msec
slat (usec): min=3, max=178, avg=14.99, stdev=10.25
clat (usec): min=34, max=954202, avg=4651.04, stdev=3734.02
lat (usec): min=46, max=954210, avg=4666.39, stdev=3733.43
clat percentiles (usec):
| 1.00th=[ 294], 5.00th=[ 1064], 10.00th=[ 1592], 20.00th=[ 2288],
| 30.00th=[ 2896], 40.00th=[ 3472], 50.00th=[ 4048], 60.00th=[ 4704],
| 70.00th=[ 5536], 80.00th=[ 6560], 90.00th=[ 8256], 95.00th=[10048],
| 99.00th=[14656], 99.50th=[17024], 99.90th=[24704], 99.95th=[29824],
| 99.99th=[57600]
lat (usec): 50=0.04%, 100=0.10%, 250=0.73%, 500=0.85%, 750=1.10%
lat (usec): 1000=1.65%
lat (msec): 2=11.10%, 4=33.54%, 10=45.68%, 20=4.96%, 50=0.23%
lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
cpu: usr=0.62%, sys=1.59%, ctx=631152, majf=0, minf=11
IO depths: 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued: total=r=0/w=2240428/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency: target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
READ: io=134300KB, aggrb=51KB/s, minb=51KB/s, maxb=51KB/s, mint=2616875msec, maxt=2616875msec
WRITE: io=8751.7MB, aggrb=3425KB/s, minb=3425KB/s, maxb=3425KB/s, mint=2616298msec, maxt=2616298msec

Disk stats (read/write):
sdb: ios=33638/2240428, merge=0/0, ticks=10456604/10286276, in_queue=20742956, util=99.89%