hadoop2.2编程：hadoop性能测试

2024-02-11 13:40:58

《hadoop the definitive way》(third version)中的Benchmarking a Hadoop Cluster Test Cases 的class在新的版本中已不再是hadoop-*-test.jar, 新版本中做BanchMark Test应采用如下方法：

1. TestDFSIO

write

TestDFSIO用来测试HDFS的I/O 性能，用一个MapReduce job来并行读取/写入文件，每个文件在一个独立的map task里被读取或写入，而map的输出用来收集该文件被执行过程中的统计数据，

写入２个文件，每个10MB

$yarn jar share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar \
TestDFSIO -write -nrFiles 2 -fileSize 10

提交job时的consol输出：

 // :: INFO fs.TestDFSIO: TestDFSIO.1.7
 // :: INFO fs.TestDFSIO: nrFiles =
 // :: INFO fs.TestDFSIO: nrBytes (MB) = 10.0
 // :: INFO fs.TestDFSIO: bufferSize =
 // :: INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
 // :: INFO fs.TestDFSIO: creating control  bytes,  files
 // :: INFO fs.TestDFSIO: created control files  files
 // :: INFO client.RMProxy: Connecting to ResourceManager at cluster1/
 // :: INFO client.RMProxy: Connecting to ResourceManager at cluster1/
 // :: INFO mapred.FileInputFormat: Total input paths to process :
 // :: INFO mapreduce.JobSubmitter: number of splits:
 // :: INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1384321503481_0003
 // :: INFO impl.YarnClientImpl: Submitted application application_1384321503481_0003 to ResourceManager at cluster1/
 // :: INFO mapreduce.Job: The url to track the job: http://cluster1:8888/proxy/application_1384321503481_0003/
 // :: INFO mapreduce.Job: Running job: job_1384321503481_0003

从consol输出可以看到:

(1)最终文件默认会被写入hdfs里的/benchmarks/TestDFSIO文件夹下， benchmarks文件夹默认位于hdfs里当前用户下面，此处位于/user/grid/下面，通过test.build.data的系统变量可以修改默认设置。

(2)2个map task (number of splits:2), 同时也证明每一个文件的写入或读取都被单独作为一个map task

job跑完后的console输出：

// :: INFO mapreduce.Job:  map % reduce %
// :: INFO mapreduce.Job: Job job_1384321503481_0003 completed successfully
// :: INFO mapreduce.Job: Counters:
    File System Counters
        FILE: Number of bytes read=
        FILE: Number of bytes written=
        FILE: Number of read operations=
        FILE: Number of large read operations=
        FILE: Number of
        HDFS: Number of bytes read=
        HDFS: Number of bytes written=
        HDFS: Number of read operations=
        HDFS: Number of large read operations=
        HDFS: Number of
    Job Counters
        Launched map tasks=
        Launched reduce tasks=
        Data-local map tasks=
        Total
        Total
    Map-Reduce Framework
        Map input records=
        Map output records=
        Map output bytes=
        Map output materialized bytes=
        Input
        Combine input records=
        Combine output records=
        Reduce input
        Reduce shuffle bytes=
        Reduce input records=
        Reduce output records=
        Spilled Records=
        Shuffled Maps =
        Failed Shuffles=
        Merged Map outputs=
        GC
        CPU
        Physical memory (bytes) snapshot=
        Virtual memory (bytes) snapshot=
        Total committed heap usage (bytes)=
    Shuffle Errors
        BAD_ID=
        CONNECTION=
        IO_ERROR=
        WRONG_LENGTH=
        WRONG_MAP=
        WRONG_REDUCE=
    File Input Format Counters
        Bytes Read=
    File Output Format Counters
        Bytes Written=
// :: INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
// :: INFO fs.TestDFSIO:            Date &  :: PST
// :: INFO fs.TestDFSIO:        Number of files:
// :: INFO fs.TestDFSIO: Total MBytes processed: 20.0
// :: INFO fs.TestDFSIO:      Throughput mb/sec: 0.5591277606933184
// :: INFO fs.TestDFSIO: Average IO rate mb/sec: 0.5635650753974915
// :: INFO fs.TestDFSIO:  IO rate std deviation: 0.05000733272172887
// :: INFO fs.TestDFSIO:     Test exec time sec: 534.566
// :: INFO fs.TestDFSIO:

从图中可以看到map task 2, reduce task 1, 统计结果中有平均I/O速率，整体速率， job运行时间，写入文件数;

read

$yarn jar \
share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient--tests.jar \
TestDFSIO -read  -nrFiles  -fileSize

就不仔细分析了，自己试试。

2. MapReduce Test with Sort

hadoop提供了一个MapReduce 程序，可以测试整个MapReduce System。此基准测试分三步：

产生random data
sort data
validate results

步骤如下:

产生random data

$yarn jar \
share/hadoop/mapreduce/hadoop-mapreduce-examples-.jar\ randomwriter random-data

用RandomWriter产生random data, 在yarn上运行RandomWriter会启动一个MapReduce job, 每个node上默认启动10个map task, 每个map 会产生1GB的random data.

修改默认参数： test.randomwriter.maps_per_host, test.randomwrite.bytes_per_map

sort data

$yarn jar \
share/hadoop/mapreduce/hadoop-mapreduce-examples-.jar \
sort random-data sorted-data
#the command 会启动一个SortValidator 程序，
#此程序会做一些列检查例如检查unsorted和sorted data是否精确

3. 其他Tests

MRBench –invoked by mrbench, 此程序会启动一个程序，运行多次
NNBench – invoked by nnbench, namenode上的负载测试
Gridmix --暂时没兴趣

（完）

码农公寓

相关文章