co.cask.cdap.kafka.flow
Class KafkaConsumerFlowlet<KEY,PAYLOAD,OFFSET>

java.lang.Object
  co.cask.cdap.api.flow.flowlet.AbstractFlowlet
      co.cask.cdap.kafka.flow.KafkaConsumerFlowlet<KEY,PAYLOAD,OFFSET>

Type Parameters:

KEY - Type of message key

PAYLOAD - Type of message value

OFFSET - Type of offset object

All Implemented Interfaces:

co.cask.cdap.api.flow.flowlet.Callback, co.cask.cdap.api.flow.flowlet.Flowlet, co.cask.cdap.api.ProgramLifecycle<co.cask.cdap.api.flow.flowlet.FlowletContext>

public abstract class KafkaConsumerFlowlet<KEY,PAYLOAD,OFFSET>
extends co.cask.cdap.api.flow.flowlet.AbstractFlowlet

Abstract base class for implementing consuming data from a Kafka cluster. This class serves as the generic base to help in implementing a flowlet that can poll from a specific Kafka version. Users should be extending from one of the sub-classes of this class.

Field Summary

protected static int

SO_TIMEOUT

Constructor Summary

KafkaConsumerFlowlet()

Method Summary

protected abstract void	configureKafka(KafkaConfigurer configurer) Configure Kafka consumer.
protected KEY	decodeKey(ByteBuffer buffer) Override this method to provide custom decoding of a message key.
protected PAYLOAD	decodePayload(ByteBuffer buffer) Override this method to provide custom decoding of a message payload.
protected abstract OFFSET	getBeginOffset(org.apache.twill.kafka.client.TopicPartition topicPartition) Returns the read offsets to start with for the given TopicPartition.
protected KafkaConfig	getKafkaConfig() Returns a Kafka configuration.
protected co.cask.cdap.api.dataset.lib.KeyValueTable	getOffsetStore() Override to return a KeyValueTable for storing consumer offsets.
protected String	getStoreKey(org.apache.twill.kafka.client.TopicPartition topicPartition) Returns the key to be used when persisting offsets into a KeyValueTable.
protected void	handleInstancesChanged(KafkaConsumerConfigurer configurer) Override to handle changes in flowlet instances.
void	initialize(co.cask.cdap.api.flow.flowlet.FlowletContext context) Initialize this Flowlet.
co.cask.cdap.api.flow.flowlet.FailurePolicy	onFailure(Object input, co.cask.cdap.api.flow.flowlet.InputContext inputContext, co.cask.cdap.api.flow.flowlet.FailureReason reason)
void	onSuccess(Object input, co.cask.cdap.api.flow.flowlet.InputContext inputContext)
void	pollMessages() A Tick method that triggered periodically by the Flow system to poll messages from Kafka.
protected void	processMessage(KafkaMessage<OFFSET> message) Overrides this method if interested in the raw Kafka message.
protected void	processMessage(KEY key, PAYLOAD payload) Override this method if interested in both the key and payload of a message read from Kafka.
protected void	processMessage(PAYLOAD payload) Override this method if only interested in the payload of a message read from Kafka.
protected abstract Iterator<KafkaMessage<OFFSET>>	readMessages(KafkaConsumerInfo<OFFSET> consumerInfo) Read messages from Kafka.
protected abstract void	saveReadOffsets(Map<org.apache.twill.kafka.client.TopicPartition,OFFSET> offsets) Persists read offsets for all topic-partition that this Flowlet consumes from Kafka.
protected void	stopService(com.google.common.util.concurrent.Service service) Stops a Service and waits for the completion.

Methods inherited from class co.cask.cdap.api.flow.flowlet.AbstractFlowlet

configure, destroy, getContext, getDescription, getName

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SO_TIMEOUT

protected static final int SO_TIMEOUT

See Also:

Constant Field Values

Constructor Detail

KafkaConsumerFlowlet

public KafkaConsumerFlowlet()

Method Detail

initialize

public void initialize(co.cask.cdap.api.flow.flowlet.FlowletContext context)
                throws Exception

Initialize this Flowlet. Child class must call this method explicitly when overriding it.

Specified by:

initialize in interface co.cask.cdap.api.flow.flowlet.Flowlet

Specified by:

initialize in interface co.cask.cdap.api.ProgramLifecycle<co.cask.cdap.api.flow.flowlet.FlowletContext>

Overrides:

initialize in class co.cask.cdap.api.flow.flowlet.AbstractFlowlet

Throws:

Exception

pollMessages

public void pollMessages()
                  throws Exception

A Tick method that triggered periodically by the Flow system to poll messages from Kafka. The default poll delay is 100 milliseconds. This method can be overridden to provide different delay value.
E.g. to override with 1 second instead:

 @Override
 @Tick(delay = 1, unit = TimeUnit.SECONDS)
 public void pollMessages() {
   super.pollMessages();
 }
 

Throws:

Exception

onSuccess

public void onSuccess(Object input,
                      co.cask.cdap.api.flow.flowlet.InputContext inputContext)

Specified by:

onSuccess in interface co.cask.cdap.api.flow.flowlet.Callback

Overrides:

onSuccess in class co.cask.cdap.api.flow.flowlet.AbstractFlowlet

onFailure

public co.cask.cdap.api.flow.flowlet.FailurePolicy onFailure(Object input,
                                                             co.cask.cdap.api.flow.flowlet.InputContext inputContext,
                                                             co.cask.cdap.api.flow.flowlet.FailureReason reason)

Specified by:

onFailure in interface co.cask.cdap.api.flow.flowlet.Callback

Overrides:

onFailure in class co.cask.cdap.api.flow.flowlet.AbstractFlowlet

getOffsetStore

protected co.cask.cdap.api.dataset.lib.KeyValueTable getOffsetStore()

Override to return a KeyValueTable for storing consumer offsets.

configureKafka

protected abstract void configureKafka(KafkaConfigurer configurer)

Configure Kafka consumer. This method will be called during the initialize(FlowletContext) phase, hence it has access to FlowletContext through the AbstractFlowlet.getContext() method.

Parameters:

configurer - for configuring consuming from Kafka

readMessages

protected abstract Iterator<KafkaMessage<OFFSET>> readMessages(KafkaConsumerInfo<OFFSET> consumerInfo)

Read messages from Kafka.

Parameters:

consumerInfo - Contains information about where to fetch messages from

Returns:

An Iterator containing sequence of messages read from Kafka. The first message must has offset no earlier than the KafkaConsumerInfo.getReadOffset() as given in the parameter.

getBeginOffset

protected abstract OFFSET getBeginOffset(org.apache.twill.kafka.client.TopicPartition topicPartition)

Returns the read offsets to start with for the given TopicPartition.

saveReadOffsets

protected abstract void saveReadOffsets(Map<org.apache.twill.kafka.client.TopicPartition,OFFSET> offsets)

Persists read offsets for all topic-partition that this Flowlet consumes from Kafka.

handleInstancesChanged

protected void handleInstancesChanged(KafkaConsumerConfigurer configurer)

Override to handle changes in flowlet instances. Sub-class may do rebalancing of topic partition that it consumes.

Parameters:

configurer - for configuring consuming from Kafka

getKafkaConfig

protected final KafkaConfig getKafkaConfig()

Returns a Kafka configuration.

processMessage

protected void processMessage(KafkaMessage<OFFSET> message)
                       throws Exception

Overrides this method if interested in the raw Kafka message.

Parameters:

message - The message fetched from Kafka.

Throws:

Exception

processMessage

protected void processMessage(KEY key,
                              PAYLOAD payload)
                       throws Exception

Override this method if interested in both the key and payload of a message read from Kafka.

Parameters:

key - Key decoded from the message

payload - Payload decoded from the message

Throws:

Exception

processMessage

protected void processMessage(PAYLOAD payload)
                       throws Exception

Override this method if only interested in the payload of a message read from Kafka.

Parameters:

payload - Payload decoded from the message

Throws:

Exception

decodeKey

protected KEY decodeKey(ByteBuffer buffer)

Override this method to provide custom decoding of a message key.

Parameters:

buffer - The bytes representing the key in the Kafka message

Returns:

The decoded key

decodePayload

protected PAYLOAD decodePayload(ByteBuffer buffer)

Override this method to provide custom decoding of a message payload.

Parameters:

buffer - The bytes representing the payload in the Kafka message

Returns:

The decoded payload

stopService

protected final void stopService(com.google.common.util.concurrent.Service service)

Stops a Service and waits for the completion. If there is exception during stop, it will get logged.

getStoreKey

protected String getStoreKey(org.apache.twill.kafka.client.TopicPartition topicPartition)

Returns the key to be used when persisting offsets into a KeyValueTable.