Flume Source官网剖析(博主推荐）

2023-02-14 12:08:45

　　不多说，直接上干货！

　　一切来源于flume官网

http://flume.apache.org/FlumeUserGuide.html

Flume Sources

Avro Source

　　官网给的例子是

a1.sources = r1

a1.channels = c1

a1.sources.r1.type = avro

a1.sources.r1.channels = c1

a1.sources.r1.bind = 0.0.0.0

a1.sources.r1.port =

　而我们常用的一般是，

agent1.sources = avro-source1

agent1.channels = ch1

#Define and configure an Spool directory source

agent1.sources.avro-source1.channels=ch1

agent1.sources.avro-source1.type=avro

agent1.sources.avro-source1.bind=0.0.0.0

agent1.sources.avro-source1.port=4141

a1.sources.r1.interceptors = i1
a1.sources.r1.interceptors.i1.type = com.djt.flume.interceptor.BehaviorIterceptor$BehaviorIterceptorBuilder

Thrift Source

Exec Source

JMS Source

Spooling Directory Source（常用）

　　官网上给的参考例子是

a1.channels = ch-

a1.sources = src-

a1.sources.src-.type = spooldir

a1.sources.src-.channels = ch-

a1.sources.src-.spoolDir = /var/log/apache/flumeSpool

a1.sources.src-.fileHeader = true

　　而我们常用的一般是，

agent1.sources = spool-source1

agent1.channels = ch1

#Define and configure an Spool directory source

agent1.sources.spool-source1.channels=ch1

agent1.sources.spool-source1.type=spooldir

agent1.sources.spool-source1.spoolDir=/home/hadoop/data/flume/sqooldir

agent1.sources.spool-source1.ignorePattern=event(_\d{}\-d{}\-d{}\_d{}\_d{})?\.log(\.COMPLETED)?

agent1.sources.spool-source1.deserializer.maxLineLength=

Taildir Source

Twitter 1% firehose Source

Kafka Source（常用）

　　官网给的例子是

Example for topic subscription by comma-separated topic list.

tier1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource

tier1.sources.source1.channels = channel1

tier1.sources.source1.batchSize =

tier1.sources.source1.batchDurationMillis =

tier1.sources.source1.kafka.bootstrap.servers = localhost:

tier1.sources.source1.kafka.topics = test1, test2

tier1.sources.source1.kafka.consumer.group.id = custom.g.id

Example for topic subscription by regex

tier1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource

tier1.sources.source1.channels = channel1

tier1.sources.source1.kafka.bootstrap.servers = localhost:

tier1.sources.source1.kafka.topics.regex = ^topic[-]$

# the default kafka.consumer.group.id=flume is used

　　具体官网里，还给了Security and Kafka Source、TLS and Kafka Source、Kerberos and Kafka Source。自行去看吧