Running through a suite of benchmark tests on the MNT Reform 2, and I’ve come across the strangest result: USB performance that bounces all over the place.
Test setup: A USB 3.0 SSD, the same one I always use, connected to one of the Reform’s USB ports. Drive is formatted ext4, mounted, and FIO is run with the following settings:
sudo fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=read --size=500m --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting
The drive’s capable of about 400MB/s, give-or-take - I last ran the benchmark on a Helios64 at 413MB/s read, and before that on a Raspberry Pi 400 at 362MB/s read.
The Reform… well, take a look:
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [R(1)][11.7%][r=109MiB/s][r=109 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [R(1)][21.7%][r=118MiB/s][r=118 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [R(1)][31.7%][r=14.0MiB/s][r=14 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [R(1)][41.7%][r=162MiB/s][r=162 IOPS][eta 00m:35s]
Jobs: 1 (f=1): [R(1)][51.7%][r=49.0MiB/s][r=49 IOPS][eta 00m:29s]
Jobs: 1 (f=1): [R(1)][61.7%][r=77.1MiB/s][r=77 IOPS][eta 00m:23s]
Jobs: 1 (f=1): [R(1)][71.7%][r=351MiB/s][r=351 IOPS][eta 00m:17s]
Jobs: 1 (f=1): [R(1)][81.7%][r=206MiB/s][r=206 IOPS][eta 00m:11s]
Jobs: 1 (f=1): [R(1)][91.7%][r=14.0MiB/s][r=14 IOPS][eta 00m:05s]
Jobs: 1 (f=0): [f(1)][100.0%][r=36.0MiB/s][r=36 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=7582: Thu Sep 9 11:33:33 2021
read: IOPS=85, BW=85.7MiB/s (89.9MB/s)(5214MiB/60819msec)
slat (usec): min=41, max=371996, avg=11271.27, stdev=41497.21
clat (msec): min=23, max=2628, avg=360.97, stdev=499.40
lat (msec): min=25, max=2752, avg=372.24, stdev=510.39
clat percentiles (msec):
| 1.00th=[ 58], 5.00th=[ 73], 10.00th=[ 79], 20.00th=[ 81],
| 30.00th=[ 83], 40.00th=[ 86], 50.00th=[ 102], 60.00th=[ 159],
| 70.00th=[ 253], 80.00th=[ 575], 90.00th=[ 1183], 95.00th=[ 1653],
| 99.00th=[ 2022], 99.50th=[ 2089], 99.90th=[ 2467], 99.95th=[ 2635],
| 99.99th=[ 2635]
bw ( KiB/s): min= 2052, max=396518, per=99.88%, avg=87680.17, stdev=102817.14, samples=121
iops : min= 2, max= 387, avg=85.59, stdev=100.35, samples=121
lat (msec) : 50=0.65%, 100=49.04%, 250=19.91%, 500=8.84%, 750=4.64%
lat (msec) : 1000=4.78%, 2000=11.05%, >=2000=1.09%
cpu : usr=0.14%, sys=1.36%, ctx=2583, majf=0, minf=554
IO depths : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.7%, 16=3.4%, 32=93.5%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
issued rwts: total=5214,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=85.7MiB/s (89.9MB/s), 85.7MiB/s-85.7MiB/s (89.9MB/s-89.9MB/s), io=5214MiB (5467MB), run=60819-60819msec
Disk stats (read/write):
sda: ios=10386/0, merge=0/0, ticks=3356212/0, in_queue=3356213, util=100.00%
It finishes at 89.9MB/s, which is way too low - but take a look at how it gets there: the highest throughput recorded is 396,518kiB/s, which is roughly what I’d expect to see; the lowest is just 2,025kiB/s.
I’ve never seen results like it. Oddly, there isn’t a single error to be seen. The only relevant lines:
[Thu Sep 9 11:27:52 2021] usb 2-1.2: new SuperSpeed Gen 1 USB device number 3 using xhci-hcd
[Thu Sep 9 11:27:52 2021] usb 2-1.2: New USB device found, idVendor=26bd, idProduct=9917, bcdDevice= 1.00
[Thu Sep 9 11:27:52 2021] usb 2-1.2: New USB device strings: Mfr=2, Product=3, SerialNumber=1
[Thu Sep 9 11:27:52 2021] usb 2-1.2: Product: Portable SSD 3.0
[Thu Sep 9 11:27:52 2021] usb 2-1.2: Manufacturer: Integral
[Thu Sep 9 11:27:52 2021] usb 2-1.2: SerialNumber: 5111905021822DD30026
[Thu Sep 9 11:27:52 2021] scsi host0: uas
[Thu Sep 9 11:27:52 2021] scsi 0:0:0:0: Direct-Access Integral Portable SSD 3.0 0 PQ: 0 ANSI: 6
[Thu Sep 9 11:27:52 2021] sd 0:0:0:0: Attached scsi generic sg0 type 0
[Thu Sep 9 11:27:52 2021] sd 0:0:0:0: [sda] Spinning up disk...
[Thu Sep 9 11:27:53 2021] .ready
[Thu Sep 9 11:27:53 2021] sd 0:0:0:0: [sda] 234441648 512-byte logical blocks: (120 GB/112 GiB)
[Thu Sep 9 11:27:53 2021] sd 0:0:0:0: [sda] Write Protect is off
[Thu Sep 9 11:27:53 2021] sd 0:0:0:0: [sda] Mode Sense: 43 00 00 00
[Thu Sep 9 11:27:53 2021] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[Thu Sep 9 11:27:53 2021] sd 0:0:0:0: [sda] Optimal transfer size 33553920 bytes
[Thu Sep 9 11:27:53 2021] sda: sda1
[Thu Sep 9 11:27:53 2021] sd 0:0:0:0: [sda] Attached SCSI disk
[Thu Sep 9 11:28:44 2021] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
Any ideas?