现在我们准备做第一个作业Lab1啦
Part I
The Map/Reduce implementation you are given is missing some pieces. Before you can write your first Map/Reduce function pair, you will need to fix the sequential
implementation. In particular, the code we give you is missing two crucial pieces: the function that divides up the output of a map task, and the function that gathers all the inputs for a reduce task. These tasks are carried out by the doMap() function in common_map.go, and the doReduce() function in common_reduce.go respectively. The comments in those files should point you in the right direction.
doMap()要求对一个文件进行map操作,并输出nReduce个intermediate files。map task会运行很多(但part1是sequential的,暂时不需要考虑加锁之类的问题),每个都有自己的jobName和mapTask作为id号。为了简化很多细节,doMap()中提供了以下函数可用:
mapF():读取指定输入文件,并返回数据内容(是一堆key-value)。mapF() is the map function provided by the application. The first argument should be the input file name, though the map function typically ignores it. The second argument should be the entire input file contents. mapF() returns a slice containing the key/value pairs for reduce; see common.go for the definition of KeyValue.
reduceName():按规则生成intermediate files的文件名。编号为r的文件负责存储ihash(key) mod nReduce==r的键值对。There is one intermediate file per reduce task. The file name includes both the map task number and the reduce task number. Use the filename generated by reduceName(jobName, mapTask, r) as the intermediate file for reduce task r. Call ihash() (see below) on each key, mod nReduce, to pick r for a key/value pair.
那么doMap()的工作就是读取文件,然后枚举每个kv键值对,送给对应的intermediate file就好啦。
doReduce()就正好反过来,要求读取nMap个intermediate file中的kv键值对,调用用户指定的reduce function(这里是把对key相同的values给append到一起),并写入outFile。reduce task也会运行很多(但part1是sequential的,暂时不需要考虑加锁之类的问题),每个都有自己的jobName和reduceTask作为id号。doReduce()同样提供了很多函数可用:
reduceName():reduceName(jobName, m, reduceTask) yields the file name from map task m.
reduceF():reduceF() is the application's reduce function. You should call it once per distinct key, with a slice of all the values for that key. reduceF() returns the reduced value for that key.
doReduce()的工作就是对map出来的intermediate files,用一个大map来合并所有的kv键值对,然后写到输出文件。输出时按key的顺序排好序输出。
代码:https://github.com/pentium3/mit6824/commit/12d1d40f71a5b0dc1c112ecc927f4c256e7da198
PartII