寒假学习进度-9(spark streaming编程初级实践)

1.安装Flume

下载地址:

http://www.apache.org/dyn/closer.lua/flume/1.7.0/apache-flume-1.7.0-bin.tar.gz

(1)解压安装包

sudo tar -zxvf apache-flume-1.7.0-bin.tar.gz -C /usr/local 
cd /usr/local 
sudo mv ./apache-flume-1.7.0-bin ./flume 
sudo chown -R hadoop:hadoop ./flume 

(2)配置环境变量

sudo vim ~/.bashrc 
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64; 
export FLUME_HOME=/usr/local/flume 
export FLUME_CONF_DIR=$FLUME_HOME/conf 
export PATH=$PATH:$FLUME_HOME/bin 

修改 flume-env.sh 配置文件:

cd /usr/local/flume/conf 
sudo cp ./flume-env.sh.template ./flume-env.sh 
sudo vim ./flume-env.sh 
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64; 

(3)查看flume版本信息

cd /usr/local/flume 
./bin/flume-ng version 

寒假学习进度-9(spark streaming编程初级实践)

2.使用 Avro 数据源测试 Flume 

寒假学习进度-9(spark streaming编程初级实践)寒假学习进度-9(spark streaming编程初级实践)

3.使用 netcat 数据源测试 Flume 

寒假学习进度-9(spark streaming编程初级实践)

寒假学习进度-9(spark streaming编程初级实践)

4.使用 Flume 作为 Spark Streaming 数据源 

------------------------------------------- 
Time: 1488029430000 ms 
------------------------------------------- 
Received 0 flume events. #这里省略了其他屏幕信息 
------------------------------------------- 
Time: 1488029432000 ms 
------------------------------------------- 
Received 8 flume events. #这里省略了其他屏幕信息 
------------------------------------------- 
Time: 1488029434000 ms 
------------------------------------------- 
Received 21 flume events. 

 

上一篇:Java基础——工厂模式


下一篇:SIX Spark Streaming 编程初级实践