Difference between revisions of "I/O sniffing"

From Linuxintro
imported>ThorstenStaerk
imported>ThorstenStaerk
 
(13 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
Sometimes, for example when [[troubleshooting]] performance problems you need to understand how your system interacts with its [[hard disk]]s. You may have used [[ioMeter]] to measure your disks' speed and [[ioStat]] and [[sar]] to gather statistics. However sometimes you need to drill down deeper into the analysis, you need to understand what happens on the block level. I call this I/O sniffing. The goal is to observe packets like this:
 +
 +
initiator XYZ requests block 4711 from device 0815
 +
initiator BLA writes block 1234 to device 9876
 +
block 4711 arrives from device 0815
 +
 +
It pays to have a good understanding about [[Bandwidth, latency and throughput]], [[block sizes]], caches, buffers, IOPS, async I/O vs sync I/O and buffered I/O vs direct I/O.
 +
 
You can do I/O sniffing using the command blktrace. blktrace will show you every request that goes to the disk.
 
You can do I/O sniffing using the command blktrace. blktrace will show you every request that goes to the disk.
  
Line 5: Line 13:
 
  # blktrace -d /dev/sdg -o - | blkparse -i -  
 
  # blktrace -d /dev/sdg -o - | blkparse -i -  
 
  [...]
 
  [...]
   <abbr title="device (major/minor number)">8,96</abbr>  <abbr title="processor">7</abbr>      <abbr title="sequence number">106</abbr>    <abbr title="timestamp">0.373952974</abbr> <abbr title="PID">11364</abbr>  <abbr title="Event">D</abbr>  <abbr title="Action">W</abbr> <abbr title="starting block + blocks">0 + 8</abbr> <abbr title="process">[kworker/7:2]</abbr>
+
   <abbr title="device (major/minor number)">8,96</abbr>  <abbr title="processor">7</abbr>      <abbr title="sequence number">106</abbr>    <abbr title="timestamp">0.373952974</abbr> <abbr title="PID">11364</abbr>  <abbr title="Action/Event">D</abbr>  <abbr title="RWBS (D) field">W</abbr> <abbr title="starting block + blocks">0 + 8</abbr> <abbr title="process">[kworker/7:2]</abbr>
 
   8,96  7      107    0.374456639    47  C  W 0 + 8 [0]
 
   8,96  7      107    0.374456639    47  C  W 0 + 8 [0]
  
  
Event can be a combination of
+
The RWBS(D) field can be a combination of
 
  R : Read
 
  R : Read
 
  W : Write
 
  W : Write
Line 15: Line 23:
 
  B : Barrier operation
 
  B : Barrier operation
 
  S : Synchronous operations
 
  S : Synchronous operations
 +
 +
= Exploring blktrace =
 +
Using the command
 +
blktrace -d /dev/sdg -o - | blkparse -i - |grep -v kworker
 +
shows me all operations but the kernel ones.
 +
 +
Now the command
 +
dd if=/dev/sdg count=1
 +
results in a lot of operations when I do it for the first time, but only "N" operations when I do it subsequently: The block is already in the files system cache. So let's omit the file system cache:
 +
dd if=/dev/sdg count=1 iflag=direct
 +
Now this results in a read operation every time I do it. iflag=direct circumvents the file system cache. With this flag it is no longer possible to set a wrong block size:
 +
# dd if=/dev/sdg count=1 iflag=direct bs=8193
 +
dd: error reading ‘/dev/sdg’: Invalid argument
 +
But it is possible to do IO in chunks of 16 blocks:
 +
dd if=/dev/sdg count=1 iflag=direct bs=8192
 +
results in
 +
8,96  6      24  218.106376166  6904  D  R 0 + 16 [dd]
 +
 +
You see here data is read in 16 sector chunks. This allows you to do '''I/O profiling''', finding out if your application does rather read or write and in which block sizes.
 +
 +
= See also =
 +
* [http://linux.die.net/man/8/blktrace blktrace man page]
 +
* [http://linux.die.net/man/1/blkparse blkparse man page]
 +
* http://unix.stackexchange.com/questions/105610/how-does-blktrace-work
 +
* http://unix.stackexchange.com/questions/105612/why-does-blktrace-only-write-blocks-of-8
 +
 +
[[Category:geeky]][[Category:low-level]][[Category:analysis]]

Latest revision as of 04:15, 19 April 2014

Sometimes, for example when troubleshooting performance problems you need to understand how your system interacts with its hard disks. You may have used ioMeter to measure your disks' speed and ioStat and sar to gather statistics. However sometimes you need to drill down deeper into the analysis, you need to understand what happens on the block level. I call this I/O sniffing. The goal is to observe packets like this:

initiator XYZ requests block 4711 from device 0815
initiator BLA writes block 1234 to device 9876 
block 4711 arrives from device 0815

It pays to have a good understanding about Bandwidth, latency and throughput, block sizes, caches, buffers, IOPS, async I/O vs sync I/O and buffered I/O vs direct I/O.

You can do I/O sniffing using the command blktrace. blktrace will show you every request that goes to the disk.

Example:

# blktrace -d /dev/sdg -o - | blkparse -i - 
[...]
  8,96   7      106     0.373952974 11364  D   W 0 + 8 [kworker/7:2]
  8,96   7      107     0.374456639    47  C   W 0 + 8 [0]


The RWBS(D) field can be a combination of

R : Read
W : Write
D : Block discard
B : Barrier operation
S : Synchronous operations

Exploring blktrace

Using the command

blktrace -d /dev/sdg -o - | blkparse -i - |grep -v kworker

shows me all operations but the kernel ones.

Now the command

dd if=/dev/sdg count=1

results in a lot of operations when I do it for the first time, but only "N" operations when I do it subsequently: The block is already in the files system cache. So let's omit the file system cache:

dd if=/dev/sdg count=1 iflag=direct

Now this results in a read operation every time I do it. iflag=direct circumvents the file system cache. With this flag it is no longer possible to set a wrong block size:

# dd if=/dev/sdg count=1 iflag=direct bs=8193
dd: error reading ‘/dev/sdg’: Invalid argument

But it is possible to do IO in chunks of 16 blocks:

dd if=/dev/sdg count=1 iflag=direct bs=8192

results in

8,96   6       24   218.106376166  6904  D   R 0 + 16 [dd]

You see here data is read in 16 sector chunks. This allows you to do I/O profiling, finding out if your application does rather read or write and in which block sizes.

See also