1. 启动Flume，出现脚本错误
错误如下：
bin/flume-ng: line 82: syntax error in conditional expression: unexpected token `('
bin/flume-ng: line 82: syntax error near `^java\.library\.path=(.'
bin/flume-ng: line 82: ` if [[ $line =~ ^java\.library\.path=(.*)$ ]]; then'
将字符串的引用，加上双引号即可。如if [[ “$line” =~ “^java\.library\.path=(.*)$” ]]; then 。
flume-ng中有82、105、154行需要修改 2. flume sink 对接hdfs
29 Feb 2016 05:57:10,372 ERROR [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcher
Runnable.run:145) - Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:38)
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:36)
at org.apache.hadoop.security.UserGroupInformation$UgiMetrics.create(UserGroupInformation.java:108)
at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:210)
at org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:553)
at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:272)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:418)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
它通过hadoop命令,自动去找到了hadoop的所有jra包并添加到了自己的classpath里面,所以就不存在这个错了.
那它到底依赖哪些jar包呢? 我也是快崩溃了,一个一个的找,一个一个的试,最终试出来了结果.
其实报错的那个类,在hadoop2的时候,是在hadoop-common当中. hadoop 1的话,应该是在hadoop-core中.
首先把这个hadoop-common包拷贝到flume的lib目录里面.
总结一下,安装完flume之后,需要从hadoop中拷贝过来的jar包有以下
commons-configuration-1.6.jar
hadoop-auth-2.5.2.jar
hadoop-common-2.5.2.jar
hadoop-hdfs-2.5.2.jar
hadoop-mapreduce-client-core-2.5.2.jar