I/O sniffing
Sometimes, for example when troubleshooting performance problems you need to understand how your system interacts with its hard disks. You may have used ioMeter to measure your disks' speed and ioStat and sar to gather statistics. However sometimes you need to drill down deeper into the analysis, you need to understand what happens on the block level. I call this I/O sniffing. The goal is to observe packets like this:
initiator XYZ requests block 4711 from device 0815 initiator BLA writes block 1234 to device 9876 block 4711 arrives from device 0815
It pays to have a good understanding about Bandwidth, latency and throughput, block sizes, caches, buffers, IOPS, async I/O vs sync I/O and buffered I/O vs direct I/O.
You can do I/O sniffing using the command blktrace. blktrace will show you every request that goes to the disk.
Example:
# blktrace -d /dev/sdg -o - | blkparse -i - [...] 8,96 7 106 0.373952974 11364 D W 0 + 8 [kworker/7:2] 8,96 7 107 0.374456639 47 C W 0 + 8 [0]
The RWBS(D) field can be a combination of
R : Read W : Write D : Block discard B : Barrier operation S : Synchronous operations
Exploring IOtrace
Using the command
blktrace -d /dev/sdg -o - | blkparse -i - |grep -v kworker
shows me all operations but the kernel ones.
Now the command
dd if=/dev/sdg count=1
results in a lot of operations when I do it for the first time, but only "N" operations when I do it subsequently: The block is already in the files system cache. So let's omit the file system cache:
dd if=/dev/sdg count=1 iflag=direct
Now this results in a read operation every time I do it.
