File format: currently SequenceFile, DataStream or CompressedStream (1)DataStream

will not compress output file and please don’t set codeC (2)CompressedStream requires set hdfs.codeC with an available codeC

Specify minimum number of replicas per HDFS block. If not specified, it comes from the

default Hadoop config in the classpath.

Format for sequence file records. One of Text or Writable. Set to Text before creating data files with Flume,

otherwise those files cannot be read by either Apache Impala (incubating) or Apache Hive.

Number of milliseconds allowed for HDFS operations, such as open, write, flush, close.

This number should be increased if many HDFS timeout operations are occurring.

Number of times the sink must try renaming a file, after initiating a close attempt. If set to 1,

this sink will not re-try a failed rename (due to, for example, NameNode or DataNode failure),

and may leave the file in an open state with a .tmp extension. If set to 0, the sink will try to rename the file until

the file is eventually renamed (there is no limit on the number of times it would try). The file may still remain open

if the close call fails but the data will be intact and in this case, the file will be closed only after a Flume restart.

Time in seconds between consecutive attempts to close a file. Each close call costs multiple RPC round-trips to the Namenode,

so setting this too low can cause a lot of load on the name node. If set to 0 or less,

the sink will not attempt to close the file if the first attempt fails, and may leave the file open or with a ”.tmp” extension.

Other possible options include avro_event or the fully-qualified class name of an implementation of

theEventSerializer.Builder interface.

Comma separate list of partition values identifying the partition to write to. May contain escape sequences.

E.g: If the table is partitioned by (continent: string, country :string, time : string) then ‘Asia,India,2014-02-26-01-21’

will indicate continent=Asia,country=India,time=2014-02-26-01-21

Hive grants a batch of transactions instead of single transactions to streaming clients like Flume.

This setting configures the number of desired transactions per Transaction Batch.

Data from all transactions in a single batch end up in a single file. Flume will write a maximum of

batchSize events in each transaction in the batch.

This setting in conjunction with batchSize provides control over the size of each file.

Note that eventually Hive will transparently compact these files into larger files.

(In seconds) Interval between consecutive heartbeats sent to Hive to keep unused transactions from expiring.

Set this value to 0 to disable heartbeats.

Serializer is responsible for parsing out field from the event and mapping them to columns in the hive table.

Choice of serializer depends upon the format of the data in the event. Supported serializers: DELIMITED and JSON

The mapping from input fields to columns in hive table. Specified as a comma separated list (no spaces) of

hive table columns names, identifying the input fields in order of their occurrence. To skip fields leave the

column name unspecified. Eg. ‘time,,ip,message’ indicates the 1st, 3rd and 4th fields in input map to time,

ip and message columns in the hive table.

(Type: character) Customizes the separator used by underlying serde. There can be a gain in efficiency if the

fields in serializer.fieldnames are in same order as table columns, the serializer.delimiter is same as the

serializer.serdeSeparator and number of fields in serializer.fieldnames is less than or equal to number of

table columns, as the fields in incoming event body do not need to be reordered to match order of table columns.

Use single quotes for special characters like ‘\t’. Ensure input fields do not contain this character.

NOTE: If serializer.delimiter is a single character, preferably set this to the same character

Amount of time (s) before the connection to the next hop is reset. This will force the Avro Sink to reconnect to the next hop.

This will allow the sink to connect to hosts behind a hardware load-balancer when news

hosts are added without having to restart the agent.

The level of compression to compress event. 0 = no compression and 1-9 is compression.

The higher the number the more compression

Set to true to enable SSL for this AvroSink. When configuring SSL, you can optionally

set a “truststore”, “truststore-password”, “truststore-type”, and specify whether to “trust-all-certs”.

If this is set to true, SSL server certificates for remote servers (Avro Sources) will not be checked.

This should NOT be used in production because it makes it easier for an attacker to execute a

man-in-the-middle attack and “listen in” on the encrypted connection.

The path to a custom Java truststore file. Flume uses the certificate authority information in this file

to determine whether the remote Avro Source’s SSL authentication credentials should be trusted.

If not specified, the default Java JSSE certificate authority files (typically “jssecacerts” or “cacerts”

in the Oracle JRE) will be used.

Amount of time (s) before the connection to the next hop is reset. This will force the Thrift Sink to reconnect to the next hop.

This will allow the sink to connect to hosts behind a hardware load-balancer when news hosts are added without having to restart the agent.

The path to a custom Java truststore file. Flume uses the certificate authority information in this file to determine whether the remote

Thrift Source’s SSL authentication credentials should be trusted. If not specified, the default Java JSSE certificate authority files (typically

“jssecacerts” or “cacerts” in the Oracle JRE) will be used.

Set to true to enable kerberos authentication. In kerberos mode, client-principal, client-keytab and server-principal are required for

successful authentication and communication to a kerberos enabled Thrift Source.

line separator (if you were to enter the default value into the config file,

then you would need to escape the backslash, like this: “\n”)

Should the sink coalesce multiple increments to a cell per batch.

This might give better performance if there are multiple increments to a limited number of cells.

Should the sink coalesce multiple increments to a cell per batch. This might give better performance

if there are multiple increments to a limited number of cells.

org.apache.flume.sink.hbase.

SimpleAsyncHbaseEventSerializer

The relative or absolute path on the local file system to the morphline configuration file.

Example: /etc/flume-ng/conf/morphline.conf

The maximum duration per flume transaction (ms). The transaction commits after this

duration or when batchSize is exceeded, whichever comes first.

org.apache.flume.sink.solr.

morphline.MorphlineHandlerImpl

This flag should be enabled for mission critical, large-scale online production systems that

need to make progress without downtime when unrecoverable exceptions occur.

Corrupt or malformed parser input data, parser bugs, and errors related to unknown

Solr schema fields produce unrecoverable exceptions.

Comma separated list of recoverable exceptions that tend to be transient,

in which case the corresponding task can be retried. Examples include network connection errors,

timeouts, etc. When the production mode flag is set to true,

the recoverable exceptions configured using this parameter will not be ignored and hence will lead to retries.

This flag should be enabled, if an unrecoverable exception is accidentally misclassified as recoverable.

This enables the sink to make progress and avoid retrying an event forever.

The name of the index which the date will be appended to. Example ‘flume’ -> ‘flume-yyyy-MM-dd’

Arbitrary header substitution is supported, eg. %{header} replaces with value of named event header

The type to index the document to, defaults to ‘log’ Arbitrary header substitution is supported,

eg. %{header} replaces with value of named event header

TTL in days, when set will cause the expired documents to be deleted automatically,

if not set documents will never be automatically deleted. TTL is accepted both in the earlier form

of integer only e.g. a1.sinks.k1.ttl = 5 and also with a qualifier ms (millisecond), s (second), m (minute),

h (hour), d (day) and w (week). Example a1.sinks.k1.ttl = 5d will set TTL to 5 days.

Followhttp://www.elasticsearch.org/guide/reference/mapping/ttl-field/ for more information.

org.apache.flume.sink.elasticsearch.

ElasticSearchLogStashEventSerializer

The ElasticSearchIndexRequestBuilderFactory or ElasticSearchEventSerializer to use.

Implementations of either class are accepted but ElasticSearchIndexRequestBuilderFactory is preferred.

If true, the Flume transaction will be commited and the writer will be

flushed on each batch of kite.batchSize records. This setting only

applies to flushable datasets. When true, it’s possible for temp files with

commited data to be left in the dataset directory. These files need to be

recovered by hand for the data to be visible to DatasetReaders.

Controls whether the sink will also sync data when committing the transaction.

This setting only applies to syncable datasets. Syncing gaurentees that data will

be written on stable storage on the remote system while flushing only gaurentees

that data has left Flume’s client buffers. When the kite.flushable.commitOnBatch

property is set to false, this property must also be set to false.

Parser that turns Flume Events into Kite entities. Valid values are avro

and the fully-qualified class name of an implementation of the EntityParser.Builder interface.

Policy that handles non-recoverable errors such as a missing Schema in the Event header.

The default value, retry, will fail the current batch and try again which matches the old behavior.

Other valid values are save, which will write the raw Event to the kite.error.dataset.uri dataset,

and the fully-qualified class name of an implementation of the FailurePolicy.Builder interface.

URI of the dataset where failed events are saved when kite.failurePolicy is set to save.

Required when the kite.failurePolicy is set to save.

List of brokers Kafka-Sink will connect to, to get the list of topic partitions

This can be a partial list of brokers, but we recommend at least two for HA.

The format is comma separated list of hostname:port

The topic in Kafka to which the messages will be published.

If this parameter is configured, messages will be published to this topic.

If the event header contains a “topic” field, the event will be published to that

topic overriding the topic configured here. Arbitrary header substitution is supported,

eg. %{header} is replaced with value of event header named “header”. (If using the substitution,

it is recommended to set “auto.create.topics.enable” property of Kafka broker to true.)

How many replicas must acknowledge a message before its considered successfully written.

Accepted values are 0 (Never wait for acknowledgement), 1 (wait for leader only), -1 (wait for all replicas)

Set this to -1 to avoid data loss in some cases of leader failure.

By default events are put as bytes onto the Kafka topic directly from the event body.

Set to true to store events as the Flume Avro binary format. Used in conjunction with

the same property on the KafkaSource or with the parseAsFlumeEvent property on the

Kafka Channel this will preserve any Flume headers for the producing side.

Specifies a Kafka partition ID (integer) for all events in this channel to be sent to,

unless overriden by partitionIdHeader. By default, if this property is not set,

events will be distributed by the Kafka Producer’s partitioner - including by key

if specified (or by a partitioner specified by kafka.partitioner.class).

When set, the sink will take the value of the field named using the value of this property from

the event header and send the message to the specified partition of the topic.

If the value represents an invalid partition, an EventDeliveryException will be thrown.

If the header value is present then this setting overrides defaultPartitionId.

When set, the sink will allow a message to be produced into a topic specified by the

topicHeaderproperty (if provided).

When set in conjunction with allowTopicOverride will produce a message

into the value of the header named using the value of this property. Care should be taken

when using in conjunction with the Kafka Source topicHeader property to avoid creating a loopback.

Set to SASL_PLAINTEXT, SASL_SSL or SSL if writing to Kafka using some level of security.

See below for additional info on secure setup.

If using SASL_PLAINTEXT, SASL_SSL or SSL refer to Kafka security for additional properties

that need to be set on producer.

These properties are used to configure the Kafka Producer. Any producer property

supported by Kafka can be used. The only requirement is to prepend the property name with the

prefixkafka.producer. For example: kafka.producer.linger.ms