场景
MapReduce Java API实例-统计单词出现频率:
https://blog.csdn.net/BADAO_LIUMANG_QIZHI/article/details/119410169
上面进行项目环境搭建的基础上。
怎样实现对下面这组数据进行排序
注:
博客:
https://blog.csdn.net/badao_liumang_qizhi
关注公众号
霸道的程序猿
获取编程相关电子书、教程推送与免费下载。
实现
输入数据格式为每行有一数值,通过MapReduce实现数据的排序功能。
利用Map阶段的Sort功能将要排序的数值作为map函数的key输出,
并在reduce函数设置一个计数器。
1、Map代码实现
package com.badao.sort; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; import java.io.IOException; import java.util.StringTokenizer; public class SortMapper extends Mapper<Object,Text,IntWritable,IntWritable> { public static IntWritable data = new IntWritable(); //map将输入中value化成IntWritable类型,作为输出的key @Override public void map(Object key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); data.set(Integer.parseInt(line)); //通过write函数写入到本地文件 context.write(data,new IntWritable(1)); } }
2、Reduce代码实现
package com.badao.sort; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Reducer; import java.io.IOException; public class SortReducer extends Reducer<IntWritable, IntWritable,IntWritable,IntWritable> { public static IntWritable linenum = new IntWritable(1); public static int i =1; @Override public void reduce(IntWritable key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { context.write(new IntWritable(i),key); ++i; } }
3、Job实现
package com.badao.sort; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.mapreduce.lib.reduce.IntSumReducer; import java.io.IOException; public class SortJob { public static void main(String[] args) throws InterruptedException, IOException, ClassNotFoundException { jobLocal(); } public static void jobLocal()throws IOException, ClassNotFoundException, InterruptedException { Configuration conf = new Configuration(); //实例化一个作业,word count是作业的名字 Job job = Job.getInstance(conf, "jobsort"); //指定通过哪个类找到对应的jar包 job.setJarByClass(SortJob.class); //为job设置Mapper类 job.setMapperClass(SortMapper.class); //为job设置reduce类 job.setReducerClass(SortReducer.class); //为job的输出数据设置key类 job.setOutputKeyClass(IntWritable.class); //为job输出设置value类 job.setOutputValueClass(IntWritable.class); //为job设置输入路径,输入路径是存在的文件夹/文件 FileInputFormat.addInputPath(job,new Path("D:\\sortData\\sort.txt")); //为job设置输出路径 FileOutputFormat.setOutputPath(job,new Path("D:\\sortdataout")); job.waitForCompletion(true); } }
运行后查看输出文件结果