Benchmarking and Stress Testing an Hadoop Cluster With TeraSort, TestDFSIO & Co.

http://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/

hadoop jar StressTesting.jar org.apache.hadoop.fs.TestDFSIO -write -nrFiles 5 -fileSize 1000

 ----- TestDFSIO ----- : write
Number of files: 5
Total MBytes processed: 5000.0
Throughput mb/sec: 12.824064099801996
Average IO rate mb/sec: 16.76740837097168
IO rate std deviation: 10.083921257977199
Test exec time sec: 260.245

hadoop jar StressTesting.jar org.apache.hadoop.fs.TestDFSIO -read -nrFiles 5 -fileSize 1000

----- TestDFSIO ----- : read
Number of files: 5
Total MBytes processed: 5000.0
Throughput mb/sec: 2.7987497425150236
Average IO rate mb/sec: 3.3441898822784424
IO rate std deviation: 1.5933975690918711
Test exec time sec: 1111.422

hadoop jar StressTesting.jar org.apache.hadoop.fs.TestDFSIO -clean