Flume NG Getting Started（Flume NG 新手入门指南）翻译

新手入门

• • Flume NG是什么？
    • 有什么改变？
  • 获得Flume NG
• • • 从源码构建
  • 配置
  • • flume-ng全局选项
    • flume-ng agent选项
    • flume-ng avro-client 选项
  • 提供反馈

Flume NG是什么？

Flume NG的目标是比Flume OG在简单性，大小和容易部署上有显著性地提高。为了实现这个目标，Flume NG将不会兼容Flume OG.我们目前在征求那些对测试Flume NG正确性、是否易用以及与其他系统的融合度有兴趣的用户的反馈信息。

有什么改变？

Flume NG（下一代）是在与Flume OG基础理念相同的情况下的一种大相径庭的实现。如果你已经对Flume很熟悉了，那么下面就是你需要知道内容。

• 你仍然需要有sources和sinks组件来做相同的事，只是现在用channels来连接。
• Channels是可插拔的和持久的。Flume NG中in-memory channel处理速度快但不支持持久化而file-based channel支持event持久化。
• There's no more logical or physical nodes. We call all physical nodes agents and agents can run zero or more sources and sinks.现在没有逻辑节点或者物理节点。我们称物理节点为agents而agents中可以运行0至多个sources和sinks。
• 没有master和不再依赖ZooKeeper。现在，Flume是依赖于一个简单的基于文件配置配置系统。
• 所有的都是插件，一些面向用户，一些针对工具和系统开发者。插件化组件包括channels、sources、sinks、interceptors、sink processors和event serializeres。

请自行阅读JIRAs文件寻找你认为重要的特性。

获得Flume NG

你可以在Flume官网中 Downloads 下载源码。如果你不打算为Flume打补丁的话，那么使用二进制文件将是简单的方式。

用源码构建

为了用源码搭建Flume NG，你必须要有git、Sun JDK1.6，Apache Maven3.x，大概90MB的磁盘空间和网络连接。

1. 检查资源

$ git clone https://git-wip-us.apache.org/repos/asf/flume.git flume

$ cd flume

$ git checkout trunk

2.编译项目

Apache Flume搭建时需要比默认配置更多的内存，我们建议你做以下的Maven 选项：

export MAVEN_OPTS="-Xms512m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=512m"

# Build the code and run the tests (note: use mvn install, not mvn package, since we deploy Jenkins SNAPSHOT jars daily, and Flume is a multi-module project)

$ mvn install

# ...or build the code without running the tests

$ mvn install -DskipTests

（需要说明的是Flume需要在构建路径下放置Google Protocol Buffers compiler来保证成功率。你可以根据here里面的介绍来下载和安装。）

这将在flume-ng-dist/target目录下生成两种类型的包，它们是

• apache-flume-ng-dist-1.4.0-SNAPSHOT-bin.tar.gz - A binary distribution of Flume, ready to run.
• apache-flume-ng-dist-1.4.0-SNAPSHOT-src.tar.gz - A source-only distribution of Flume.

如果你只是一个想要运行Flume的用户，你大概只需要-bin 版本。将它复制并解压，你就可以使用Flume了。

$ cp flume-ng-dist/target/apache-flume-1.4.0-SNAPSHOT-bin.tar.gz .

$ tar -zxvf apache-flume-1.4.0-SNAPSHOT-bin.tar.gz

$ cd apache-flume-1.4.0-SNAPSHOT-bin

3.按照工作模板创建一个你自己的属性文档（或者从头开始创建一个）

$ cp conf/flume-conf.properties.template conf/flume.conf

4.（可选）按照模板创建一个你自己的flume-env.sh文档（或者从头开始创建一个）。如果命令行中通过 –conf/-c指定了conf目录的话，那么fluem-ng会在该路径下搜寻“flume-env.sh”文档。一个使用flume-env.sh的情况是当你使用你自定义的Flume NG组件进行开发时通过JAVA_OPTS来指定调试或者评测选项。

$ cp conf/flume-env.sh.template conf/flume-env.sh

5.配置和运行Flume NG

在你完成 Flume NG的配置之后，你可以通过 bin/flume-ng 可执行文件来运行它。这个脚本拥有一系列参数和模式。

配置

Flume使用基于Java属性文件格式的配置文档。运行agent时需要通过 –f<file>选项来告诉Flume。这个文件可以放在任意位置，但从历史和未来的角度来看，这个conf目录将会是配置文件的正确位置。

让我们从一个基础的例子开始吧。将下面复制并粘贴到conf/flume.conf中：

# Define a memory channel called ch1 on agent1

agent1.channels.ch1.type = memory

# Define an Avro source called avro-source1 on agent1 and tell it

# to bind to 0.0.0.0:41414. Connect it to channel ch1.

agent1.sources.avro-source1.channels = ch1

agent1.sources.avro-source1.type = avro

agent1.sources.avro-source1.bind = 0.0.0.0

agent1.sources.avro-source1.port = 41414

# Define a logger sink that simply logs all events it receives

# and connect it to the other end of the same channel.

agent1.sinks.log-sink1.channel = ch1

agent1.sinks.log-sink1.type = logger

# Finally, now that we've defined all of our components, tell

# agent1 which ones we want to activate.

agent1.channels = ch1

agent1.sources = avro-source1

agent1.sinks = log-sink1

这个例子创建了内存channel（一个不可靠和高效传输）将一个Avro RPC source,和一个logger sink连接在一起。Avro source接收到的任何event都会按照规划的路线传到ch1 channel中然后传递给logger sink。需要重点说明的是定义组件只是配置Flume的前半部分工作；它们必须配置在<agent>中被激活。多个source、channel和sink是可以被配置的，用空格隔开每个组件就可以。

至于所有的鞋机，请自行查看Javadoc中的org.apache.flume.conf.properties.PropertiesFileConfigurationProvider类。

这是当前已经实现的channels、sinks和sources的列表。每个插件都有其自己的选项和需要配置的属性，请自行阅读javadoc。

flume-ng可执行可以让你运行一个Flume NG agent或者一个 Avro 客户端用于测试和实验。无论怎样，你必须指定一个命令（例如 agent或者avro-client）和一个conf目录（--conf<conf dir>）。所有其他的选项都可以用命令行指定。

使用上面的fiume.conf来启动flume服务器

bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -Dflume.root.logger=DEBUG,console -n agent1

需要注意的是agent 的名字是通过 –n agent1来指定的并且必须和-conf/flume.conf中给定的名字相匹配。

你的输出应该是这样的：

$ bin/flume-ng agent --conf conf/ -f conf/flume.conf -n agent1

2012-03-16 16:36:11,918 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:58)] Starting lifecycle supervisor 1

2012-03-16 16:36:11,921 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node starting - agent1

2012-03-16 16:36:11,926 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:110)] Node manager starting

2012-03-16 16:36:11,928 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:58)] Starting lifecycle supervisor 10

2012-03-16 16:36:11,929 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:114)] Node manager started

2012-03-16 16:36:11,926 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)] Configuration provider starting

2012-03-16 16:36:11,930 (lifecycleSupervisor-1-1) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:87)] Configuration provider started

2012-03-16 16:36:11,930 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:189)] Checking file:conf/flume.conf for changes

2012-03-16 16:36:11,931 (conf-file-poller-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:196)] Reloading configuration file:conf/flume.conf

2012-03-16 16:36:11,936 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.properties.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:225)] Starting validation of configuration for agent: agent1, initial-configuration: AgentConfiguration[agent1]

SOURCES: {avro-source1=ComponentConfiguration[avro-source1]

  CONFIG: {port=41414, channels=ch1, type=avro, bind=0.0.0.0}

  RUNNER:   ComponentConfiguration[runner]

    CONFIG: {}

}

CHANNELS: {ch1=ComponentConfiguration[ch1]

  CONFIG: {type=memory}

}

SINKS: {log-sink1=ComponentConfiguration[log-sink1]

  CONFIG: {type=logger, channel=ch1}

  RUNNER:   ComponentConfiguration[runner]

    CONFIG: {}

}

2012-03-16 16:36:11,936 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:119)] Post-validation flume configuration contains configuation  for agents: [agent1]

2012-03-16 16:36:11,937 (conf-file-poller-0) [DEBUG - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:67)] Creating instance of channel ch1 type memory

2012-03-16 16:36:11,944 (conf-file-poller-0) [DEBUG - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:73)] Creating instance of source avro-source1, type avro

2012-03-16 16:36:11,957 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:69)] Creating instance of sink log-sink1 typelogger

2012-03-16 16:36:11,963 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.onNodeConfigurationChanged(DefaultLogicalNodeManager.java:52)] Node configuration change:{ sourceRunners:{avro-source1=EventDrivenSourceRunner: { source:AvroSource: { bindAddress:0.0.0.0 port:41414 } }} sinkRunners:{log-sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@79f6f296 counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel@43b09468} }

2012-03-16 16:36:11,974 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.source.AvroSource.start(AvroSource.java:122)] Avro source starting:AvroSource: { bindAddress:0.0.0.0 port:41414 }

2012-03-16 16:36:11,975 (Thread-1) [DEBUG - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:123)] Polling sink runner starting

2012-03-16 16:36:12,352 (lifecycleSupervisor-1-1) [DEBUG - org.apache.flume.source.AvroSource.start(AvroSource.java:132)] Avro source started
 

flume-ng 全局选项

flume-ng agent选项

当给一个agent命令时，Flume NG agent将会根据一个给定的配置文件进行启动。

flume-ng avro-client 选项

运行一个Avro client从标准输入发送数据或文件到一个Flume NG Avro Source所监听的主机和端口上。

Avroclient将每一行以\n, \r, or \r\n结尾的数据当做一个event。把avro-clinet 命令当做Flume中cat命令。例如，下面的命令创建了一个event并将它发送到Flume’avro source所监听的端口41414。

在一个新的窗口输入以下内容：

$ bin/flume-ng avro-client --conf conf -H localhost -p 41414 -F /etc/passwd -Dflume.root.logger=DEBUG,console

你能看到以下信息：

2012-03-16 16:39:17,124 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:175)] Finished

2012-03-16 16:39:17,127 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:178)] Closing reader

2012-03-16 16:39:17,127 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:183)] Closing transceiver

2012-03-16 16:39:17,129 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:73)] Exiting

服务器所运行的第一个窗口会打印以下内容：

2012-03-16 16:39:16,738 (New I/O server boss #1 ([id: 0x49e808ca, /0:0:0:0:0:0:0:0:41414])) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /1

27.0.0.1:39577 => /127.0.0.1:41414] OPEN

2012-03-16 16:39:16,742 (New I/O server worker #1-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /127.0.0.1:39577 => /127.0.0.1:41414] BOU

ND: /127.0.0.1:41414

2012-03-16 16:39:16,742 (New I/O server worker #1-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /127.0.0.1:39577 => /127.0.0.1:41414] CON

NECTED: /127.0.0.1:39577

2012-03-16 16:39:17,129 (New I/O server worker #1-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /127.0.0.1:39577 :> /127.0.0.1:41414] DISCONNECTED

2012-03-16 16:39:17,129 (New I/O server worker #1-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /127.0.0.1:39577 :> /127.0.0.1:41414] UNBOUND

2012-03-16 16:39:17,129 (New I/O server worker #1-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /127.0.0.1:39577 :> /127.0.0.1:41414] CLOSED

2012-03-16 16:39:17,302 (Thread-1) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:68)] Event: { headers:{} body:[B@5c1ae90c }

2012-03-16 16:39:17,302 (Thread-1) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:68)] Event: { headers:{} body:[B@6aba4211 }

2012-03-16 16:39:17,302 (Thread-1) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:68)] Event: { headers:{} body:[B@6a47a0d4 }

2012-03-16 16:39:17,302 (Thread-1) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:68)] Event: { headers:{} body:[B@48ff4cf }

...

恭喜！你的Apache Flume已经成功运行了。

提供反馈

对于帮助构建、配置和运行Flume来说，最好的地方就是用户的邮件列表。发送邮件到user-subscribe@flume.apache.org 进行订阅和一旦你订阅之后user@flume.apache.org 会发送订阅信息给你。档案信息可以在 http://mail-archives.apache.org/mod_mbox/incubator-flume-user/ (2012/7建立) and http://mail-archives.apache.org/mod_mbox/incubator-flume-user/http://mail-archives.apache.org/mod_mbox/flume-user/ 得到。

如果你确认你发现一个bug或者需要一个特性或者提升，请不要害羞。去这个 https://issues.apache.org/jira/browse/FLUME 网站为该版本的Flume提一个JIRA.对于NG版本，请为合适的里程碑/发布留下“影响版本”的标识。可以只留下你对于未达之处的无法确定的任何想法。当我们需要的时候会像你征求细节。需要说明的是你必须创建一个Apache JIRA账户以致你可以提出问题。

下面为原文

Getting Started

• What is Flume NG?
• What's Changed?
• Getting Flume NG
• Building From Source
• Configuration （
• flume-ng global options
• flume-ng agent options（
• flume-ng avro-client options
• Providing Feedback

What is Flume NG?

Flume NG aims to be significantly simpler, smaller, and easier to deploy than Flume OG. In doing so, we do not commit to maintaining backward compatibility of Flume NG with Flume OG. We're currently soliciting feedback from those who are interested in testing Flume NG for correctness, ease of use, and potential integration with other systems.

What's Changed?

Flume NG (Next Generation) is a huge departure from Flume OG (Original Generation) in its implementation although many of the original concepts are the same. If you're already familiar with Flume, here's what you need to know.

• You still have sources and sinks and they still do the same thing. They are now connected by channels.
• Channels are pluggable and dictate durability. Flume NG ships with an in-memory channel for fast, but non-durable event delivery and a file-based channel for durable event delivery.
• There's no more logical or physical nodes. We call all physical nodes agents and agents can run zero or more sources and sinks.
• There's no master and no ZooKeeper dependency anymore. At this time, Flume runs with a simple file-based configuration system.
• Just about everything is a plugin, some end user facing, some for tool and system developers. Pluggable components include channels, sources, sinks, interceptors, sink processors, and event serializers.

Please file JIRAs and/or vote for features you feel are important.

Getting Flume NG

Flume is available as a source tarball and binary on the Downloads section of the Flume Website. If you are not planning on creating patches for Flume, the binary is likely the easiest way to get started.

Building From Source

To build Flume NG from source, you'll need git, the Sun JDK 1.6, Apache Maven 3.x, about 90MB of local disk space and an Internet connection.

1. Check out the source

$ git clone https://git-wip-us.apache.org/repos/asf/flume.git flume

$ cd flume

$ git checkout trunk

2. Compile the project

The Apache Flume build requires more memory than the default configuration. We recommend you set the following Maven options:

export MAVEN_OPTS="-Xms512m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=512m"

# Build the code and run the tests (note: use mvn install, not mvn package, since we deploy Jenkins SNAPSHOT jars daily, and Flume is a multi-module project)

$ mvn install

# ...or build the code without running the tests

$ mvn install -DskipTests

(Please note that Flume requires that Google Protocol Buffers compiler be in the path for the build to be successful. You download and install it by following the instructions here.)

This produces two types of packages in flume-ng-dist/target. They are:

• apache-flume-ng-dist-1.4.0-SNAPSHOT-bin.tar.gz - A binary distribution of Flume, ready to run.
• apache-flume-ng-dist-1.4.0-SNAPSHOT-src.tar.gz - A source-only distribution of Flume.

If you're a user and you just want to run Flume, you probably want the -bin version. Copy one out, decompress it, and you're ready to go.

$ cp flume-ng-dist/target/apache-flume-1.4.0-SNAPSHOT-bin.tar.gz .

$ tar -zxvf apache-flume-1.4.0-SNAPSHOT-bin.tar.gz

$ cd apache-flume-1.4.0-SNAPSHOT-bin

3. Create your own properties file based on the working template (or create one from scratch)

$ cp conf/flume-conf.properties.template conf/flume.conf

4. (Optional) Create your flume-env.sh file based on the template (or create one from scratch). The flume-ng executable looks for and sources a file named "flume-env.sh" in the conf directory specified by the --conf/-c commandline option. One use case for using flume-env.sh would be to specify debugging or profiling options via JAVA_OPTS when developing your own custom Flume NG components such as sources and sinks.

$ cp conf/flume-env.sh.template conf/flume-env.sh

5. Configure and Run Flume NG

After you've configured Flume NG (see below), you can run it with the bin/flume-ng executable. This script has a number of arguments and modes.

Configuration

Flume uses a Java property file based configuration format. It is required that you tell Flume which file to use by way of the -f <file> option (see above) when running an agent. The file can live anywhere, but historically - and in the future - the conf directory will be the correct place for config files.

Let's start with a basic example. Copy and paste this into conf/flume.conf:

# Define a memory channel called ch1 on agent1

agent1.channels.ch1.type = memory

# Define an Avro source called avro-source1 on agent1 and tell it

# to bind to 0.0.0.0:41414. Connect it to channel ch1.

agent1.sources.avro-source1.channels = ch1

agent1.sources.avro-source1.type = avro

agent1.sources.avro-source1.bind = 0.0.0.0

agent1.sources.avro-source1.port = 41414

# Define a logger sink that simply logs all events it receives

# and connect it to the other end of the same channel.

agent1.sinks.log-sink1.channel = ch1

agent1.sinks.log-sink1.type = logger

# Finally, now that we've defined all of our components, tell

# agent1 which ones we want to activate.

agent1.channels = ch1

agent1.sources = avro-source1

agent1.sinks = log-sink1

This example creates a memory channel (i.e. an unreliable or "best effort" transport), an Avro RPC source, and a logger sink and connects them together. Any events received by the Avro source are routed to the channel ch1 and delivered to the logger sink. It's important to note that defining components is the first half of configuring Flume; they must be activated by listing them in the <agent>.channels, <agent>.sources, and sections. Multiple sources, sinks, and channels may be listed, separated by a space.

For full details, please see the javadoc for the org.apache.flume.conf.properties.PropertiesFileConfigurationProvider class.

This is a listing of the implemented sources, sinks, and channels at this time. Each plugin has its own optional and required configuration properties so please see the javadocs (for now).

The flume-ng executable lets you run a Flume NG agent or an Avro
client which is useful for testing and experiments. No matter what, you'll need
to specify a command (e.g. agent or avro-client) and
a conf directory (--conf <conf dir>). All other options are
command-specific.

To start the flume server using the flume.conf above:

bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -Dflume.root.logger=DEBUG,console -n agent1

Notice that the agent name is specified by -n agent1 and must match a agent name given in -f conf/flume.conf

Your output should look something like this:

$ bin/flume-ng agent --conf conf/ -f conf/flume.conf -n agent1

2012-03-16 16:36:11,918 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:58)] Starting lifecycle supervisor 1

2012-03-16 16:36:11,921 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node starting - agent1

2012-03-16 16:36:11,926 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:110)] Node manager starting

2012-03-16 16:36:11,928 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:58)] Starting lifecycle supervisor 10

2012-03-16 16:36:11,929 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:114)] Node manager started

2012-03-16 16:36:11,926 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)] Configuration provider starting

2012-03-16 16:36:11,930 (lifecycleSupervisor-1-1) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:87)] Configuration provider started

2012-03-16 16:36:11,930 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:189)] Checking file:conf/flume.conf for changes

2012-03-16 16:36:11,931 (conf-file-poller-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:196)] Reloading configuration file:conf/flume.conf

2012-03-16 16:36:11,936 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.properties.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:225)] Starting validation of configuration for agent: agent1, initial-configuration: AgentConfiguration[agent1]

SOURCES: {avro-source1=ComponentConfiguration[avro-source1]

  CONFIG: {port=41414, channels=ch1, type=avro, bind=0.0.0.0}

  RUNNER:   ComponentConfiguration[runner]

    CONFIG: {}

}

CHANNELS: {ch1=ComponentConfiguration[ch1]

  CONFIG: {type=memory}

}

SINKS: {log-sink1=ComponentConfiguration[log-sink1]

  CONFIG: {type=logger, channel=ch1}

  RUNNER:   ComponentConfiguration[runner]

    CONFIG: {}

}

2012-03-16 16:36:11,936 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:119)] Post-validation flume configuration contains configuation  for agents: [agent1]

2012-03-16 16:36:11,937 (conf-file-poller-0) [DEBUG - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:67)] Creating instance of channel ch1 type memory

2012-03-16 16:36:11,944 (conf-file-poller-0) [DEBUG - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:73)] Creating instance of source avro-source1, type avro

2012-03-16 16:36:11,957 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:69)] Creating instance of sink log-sink1 typelogger

2012-03-16 16:36:11,963 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.onNodeConfigurationChanged(DefaultLogicalNodeManager.java:52)] Node configuration change:{ sourceRunners:{avro-source1=EventDrivenSourceRunner: { source:AvroSource: { bindAddress:0.0.0.0 port:41414 } }} sinkRunners:{log-sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@79f6f296 counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel@43b09468} }

2012-03-16 16:36:11,974 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.source.AvroSource.start(AvroSource.java:122)] Avro source starting:AvroSource: { bindAddress:0.0.0.0 port:41414 }

2012-03-16 16:36:11,975 (Thread-1) [DEBUG - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:123)] Polling sink runner starting

2012-03-16 16:36:12,352 (lifecycleSupervisor-1-1) [DEBUG - org.apache.flume.source.AvroSource.start(AvroSource.java:132)] Avro source started

flume-ng global options

flume-ng agent options

When given the agent command, a Flume NG agent will be started with a given configuration file (required).

flume-ng avro-client options（flume-ng avro-client 选项）

Run an Avro client that sends either a file or data from stdin to a specified host and port where a Flume NG Avro Source is listening.

The Avro client treats each line (terminated by \n, \r, or \r\n) as an event. Think of the avro-client command as cat for Flume. For instance, the following creates one event per Linux user and sends it to Flume's avro source on localhost:41414.

In a new window type the following:

$ bin/flume-ng avro-client --conf conf -H localhost -p 41414 -F /etc/passwd -Dflume.root.logger=DEBUG,console

You should see something like this:

2012-03-16 16:39:17,124 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:175)] Finished

2012-03-16 16:39:17,127 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:178)] Closing reader

2012-03-16 16:39:17,127 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:183)] Closing transceiver

2012-03-16 16:39:17,129 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:73)] Exiting

And in your first window, where the server is running:

2012-03-16 16:39:16,738 (New I/O server boss #1 ([id: 0x49e808ca, /0:0:0:0:0:0:0:0:41414])) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /1

27.0.0.1:39577 => /127.0.0.1:41414] OPEN

2012-03-16 16:39:16,742 (New I/O server worker #1-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /127.0.0.1:39577 => /127.0.0.1:41414] BOU

ND: /127.0.0.1:41414

2012-03-16 16:39:16,742 (New I/O server worker #1-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /127.0.0.1:39577 => /127.0.0.1:41414] CON

NECTED: /127.0.0.1:39577

2012-03-16 16:39:17,129 (New I/O server worker #1-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /127.0.0.1:39577 :> /127.0.0.1:41414] DISCONNECTED

2012-03-16 16:39:17,129 (New I/O server worker #1-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /127.0.0.1:39577 :> /127.0.0.1:41414] UNBOUND

2012-03-16 16:39:17,129 (New I/O server worker #1-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:123)] [id: 0x0b92a848, /127.0.0.1:39577 :> /127.0.0.1:41414] CLOSED

2012-03-16 16:39:17,302 (Thread-1) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:68)] Event: { headers:{} body:[B@5c1ae90c }

2012-03-16 16:39:17,302 (Thread-1) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:68)] Event: { headers:{} body:[B@6aba4211 }

2012-03-16 16:39:17,302 (Thread-1) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:68)] Event: { headers:{} body:[B@6a47a0d4 }

2012-03-16 16:39:17,302 (Thread-1) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:68)] Event: { headers:{} body:[B@48ff4cf }

...

Congratulations! You have Apache Flume running!

Providing Feedback

For help building, configuring, and running Flume (NG or otherwise), the best place is always the user mailing list. Send an email to user-subscribe@flume.apache.org to subscribe and user@flume.apache.org to post once you've subscribed. The archives are available at http://mail-archives.apache.org/mod_mbox/incubator-flume-user/ (up through part of July 2012) and http://mail-archives.apache.org/mod_mbox/incubator-flume-user/http://mail-archives.apache.org/mod_mbox/flume-user/ (starting through part of July 2012 onwards).

If you believe you've found a bug or wish to file a feature request or improvement, don't be shy. Go to https://issues.apache.org/jira/browse/FLUME and file a JIRA for the version of Flume. For NG, please set the "Affects Version" to the appropriate milestone / release. Just leave any field you're not sure about blank. We'll bug you for details if we need them. Note that you must create an Apache JIRA account and log in before you can file issues.

因个人能力实在有限，难免会出现这样那样的话，希望大家不吝指教。