org.apache.flume.source.kafka

Class KafkaSource

java.lang.Object

org.apache.flume.source.BasicSourceSemantics

org.apache.flume.source.AbstractPollableSource

org.apache.flume.source.kafka.KafkaSource

All Implemented Interfaces:

Configurable, LifecycleAware, NamedComponent, PollableSource, Source

public class KafkaSource
extends AbstractPollableSource
implements Configurable

A Source for Kafka which reads messages from a kafka topic. zookeeperConnect: Kafka's zookeeper connection string. Required

groupId: the group ID of consumer group. Required

topic: the topic to consume messages from. Required

maxBatchSize: Maximum number of messages written to Channel in one batch. Default: 1000

maxBatchDurationMillis: Maximum number of milliseconds before a batch (of any size) will be written to a channel. Default: 1000

kafka.auto.commit.enable: If true, commit automatically every time period. if false, commit on each batch. Default: false

kafka.consumer.timeout.ms: Polling interval for new data for batch. Low value means more CPU usage. High value means the time.upper.limit may be missed. Default: 10 Any property starting with "kafka" will be passed to the kafka consumer So you can use any configuration supported by Kafka 0.8.1.1

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flume.PollableSource

PollableSource.Status

Constructor Summary

Constructors

Constructor and Description
KafkaSource()

Method Summary

Methods

Modifier and Type	Method and Description
protected void	doConfigure(Context context) We configure the source and generate properties for the Kafka Consumer Kafka Consumer properties are generated as follows: 1.
protected PollableSource.Status	doProcess()
protected void	doStart()
protected void	doStop()

Methods inherited from class org.apache.flume.source.AbstractPollableSource

configure, getBackOffSleepIncrement, getMaxBackOffSleepInterval, process

Methods inherited from class org.apache.flume.source.BasicSourceSemantics

getChannelProcessor, getLifecycleState, getName, getStartException, isStarted, setChannelProcessor, setLifecycleState, setName, start, stop, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.flume.conf.Configurable

configure

Methods inherited from interface org.apache.flume.Source

getChannelProcessor, setChannelProcessor

Methods inherited from interface org.apache.flume.lifecycle.LifecycleAware

getLifecycleState, start, stop

Methods inherited from interface org.apache.flume.NamedComponent

getName, setName

Constructor Detail

KafkaSource

public KafkaSource()

Method Detail

doProcess

protected PollableSource.Status doProcess()
                                   throws EventDeliveryException

Specified by:

doProcess in class AbstractPollableSource

Throws:

EventDeliveryException

doConfigure

protected void doConfigure(Context context)
                    throws FlumeException

We configure the source and generate properties for the Kafka Consumer Kafka Consumer properties are generated as follows: 1. Generate a properties object with some static defaults that can be overridden by Source configuration 2. We add the configuration users added for Kafka (parameters starting with kafka. and must be valid Kafka Consumer properties 3. We add the source documented parameters which can override other properties

Specified by:

doConfigure in class BasicSourceSemantics

Parameters:

context -

Throws:

FlumeException

doStart

protected void doStart()
                throws FlumeException

Specified by:

doStart in class BasicSourceSemantics

Throws:

FlumeException

doStop

protected void doStop()
               throws FlumeException

Specified by:

doStop in class BasicSourceSemantics

Throws:

FlumeException