org.springframework.cloud.stream.binder.kafka

Class KafkaMessageChannelBinder

java.lang.Object

org.springframework.cloud.stream.binder.MessageChannelBinderSupport

org.springframework.cloud.stream.binder.kafka.KafkaMessageChannelBinder

All Implemented Interfaces:

org.springframework.beans.factory.Aware, org.springframework.beans.factory.InitializingBean, Binder<org.springframework.messaging.MessageChannel>, org.springframework.context.ApplicationContextAware

public class KafkaMessageChannelBinder
extends MessageChannelBinderSupport

A binder that uses Kafka as the underlying middleware. The general implementation mapping between XD concepts and Kafka concepts is as follows: A binder that uses Kafka as the underlying middleware. The general implementation mapping between XD concepts and Kafka concepts is as follows:

Stream definition	Kafka topic	Kafka partitions	Notes
foo = "http | log"	foo.0	1 partition	1 producer, 1 consumer
foo = "http | log", log.count=x	foo.0	x partitions	1 producer, x consumers with static group 'springXD', achieves queue semantics
foo = "http | log", log.count=x + XD partitioning	still 1 topic 'foo.0'	x partitions + use key computed by XD	1 producer, x consumers with static group 'springXD', achieves queue semantics
foo = "http | log", log.count=x, concurrency=y	foo.0	x*y partitions	1 producer, x XD consumers, each with y threads
foo = "http | log", log.count=0, x actual log containers	foo.0	10(configurable) partitions	1 producer, x XD consumers. Can't know the number of partitions beforehand, so decide a number that better be greater than number of containers

Author:

Eric Bottard, Marius Bogoevici, Ilayaperumal Gopinathan, David Turanski, Gary Russell, Mark Fisher

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	KafkaMessageChannelBinder.Mode
static class	KafkaMessageChannelBinder.StartOffset

Nested classes/interfaces inherited from class org.springframework.cloud.stream.binder.MessageChannelBinderSupport

MessageChannelBinderSupport.DirectHandler, MessageChannelBinderSupport.PartitioningMetadata, MessageChannelBinderSupport.SetBuilder, MessageChannelBinderSupport.SharedChannelProvider<T extends org.springframework.messaging.MessageChannel>

Field Summary

Fields

Modifier and Type	Field and Description
static String	AUTO_COMMIT_ENABLED
static org.apache.kafka.common.serialization.ByteArraySerializer	BYTE_ARRAY_SERIALIZER
static String	COMPRESSION_CODEC
static String	FETCH_SIZE
static int	METADATA_VERIFICATION_MAX_INTERVAL
static int	METADATA_VERIFICATION_RETRY_ATTEMPTS
static double	METADATA_VERIFICATION_RETRY_BACKOFF_MULTIPLIER
static int	METADATA_VERIFICATION_RETRY_INITIAL_INTERVAL
protected static Set<Object>	PRODUCER_COMPRESSION_PROPERTIES
static String	QUEUE_SIZE
static String	REQUIRED_ACKS
static org.I0Itec.zkclient.serialize.ZkSerializer	utf8Serializer Used when writing directly to ZK.

Fields inherited from class org.springframework.cloud.stream.binder.MessageChannelBinderSupport

CONSUMER_RETRY_PROPERTIES, CONSUMER_STANDARD_PROPERTIES, defaultBackOffInitialInterval, defaultBackOffMaxInterval, defaultBackOffMultiplier, defaultBatchBufferLimit, defaultBatchingEnabled, defaultBatchSize, defaultBatchTimeout, defaultCompress, defaultConcurrency, defaultDurableSubscription, defaultMaxAttempts, directChannelProvider, evaluationContext, logger, MEDIATYPES_MEDIATYPE_ALL, PARTITION_HEADER, PRODUCER_BATCHING_ADVANCED_PROPERTIES, PRODUCER_BATCHING_BASIC_PROPERTIES, PRODUCER_PARTITIONING_PROPERTIES, PRODUCER_STANDARD_PROPERTIES

Constructor Summary

Constructors

Constructor and Description
KafkaMessageChannelBinder(org.springframework.integration.kafka.support.ZookeeperConnect zookeeperConnect, String brokers, String zkAddress, String... headersToMap)

Method Summary

Modifier and Type	Method and Description
void	afterPropertiesSet()
Binding<org.springframework.messaging.MessageChannel>	bindProducer(String name, org.springframework.messaging.MessageChannel moduleOutputChannel, Properties properties)
org.springframework.integration.kafka.listener.KafkaMessageListenerContainer	createMessageListenerContainer(Properties properties, String group, int maxConcurrency, String topic, long referencePoint)
protected Binding<org.springframework.messaging.MessageChannel>	doBindConsumer(String name, String group, org.springframework.messaging.MessageChannel inputChannel, Properties properties)
void	doManualAck(LinkedList<org.springframework.messaging.MessageHeaders> messageHeadersList)
org.springframework.integration.kafka.core.ConnectionFactory	getConnectionFactory()
KafkaMessageChannelBinder.StartOffset	getStartOffset()
boolean	isResetOffsets()
void	setDefaultAutoCommitEnabled(boolean defaultAutoCommitEnabled) Set the default auto commit enabled property; This is used to commit the offset either automatically or manually.
void	setDefaultCompressionCodec(String defaultCompressionCodec)
void	setDefaultFetchSize(int defaultFetchSize)
void	setDefaultMaxWait(int defaultMaxWait)
void	setDefaultMinPartitionCount(int defaultMinPartitionCount)
void	setDefaultQueueSize(int defaultQueueSize)
void	setDefaultReplicationFactor(int defaultReplicationFactor)
void	setDefaultRequiredAcks(int defaultRequiredAcks)
void	setMode(KafkaMessageChannelBinder.Mode mode)
void	setOffsetStoreBatchBytes(int offsetStoreBatchBytes)
void	setOffsetStoreBatchTime(int offsetStoreBatchTime)
void	setOffsetStoreMaxFetchSize(int offsetStoreMaxFetchSize)
void	setOffsetStoreRequiredAcks(int offsetStoreRequiredAcks)
void	setOffsetStoreRetentionTime(int offsetStoreRetentionTime)
void	setOffsetStoreSegmentSize(int offsetStoreSegmentSize)
void	setOffsetStoreTopic(String offsetStoreTopic)
void	setOffsetUpdateCount(int offsetUpdateCount)
void	setOffsetUpdateShutdownTimeout(int offsetUpdateShutdownTimeout)
void	setOffsetUpdateTimeWindow(int offsetUpdateTimeWindow)
void	setResetOffsets(boolean resetOffsets)
void	setRetryOperations(org.springframework.retry.RetryOperations retryOperations) Retry configuration for operations such as validating topic creation
void	setSocketBufferSize(int socketBufferSize)
void	setStartOffset(KafkaMessageChannelBinder.StartOffset startOffset)
static void	validateTopicName(String topicName) Allowed chars are ASCII alphanumerics, '.', '_' and '-'.

Methods inherited from class org.springframework.cloud.stream.binder.MessageChannelBinderSupport

addBinding, afterUnbind, applyPrefix, bindConsumer, buildPartitionRoutingExpression, buildRetryTemplateIfRetryEnabled, constructDLQName, deserializePayloadIfNecessary, deserializePayloadIfNecessary, determinePartition, doBindDynamicProducer, getApplicationContext, getBeanFactory, getIdGenerator, groupedName, serializePayloadIfNecessary, setApplicationContext, setCodec, setDefaultBackOffInitialInterval, setDefaultBackOffMaxInterval, setDefaultBackOffMultiplier, setDefaultBatchBufferLimit, setDefaultBatchingEnabled, setDefaultBatchSize, setDefaultBatchTimeout, setDefaultCompress, setDefaultConcurrency, setDefaultDurableSubscription, setDefaultMaxAttempts, setIntegrationEvaluationContext, setPartitionSelector, stopBindings, unbind, validateConsumerProperties, validateProducerProperties

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

BYTE_ARRAY_SERIALIZER

public static final org.apache.kafka.common.serialization.ByteArraySerializer BYTE_ARRAY_SERIALIZER

METADATA_VERIFICATION_RETRY_ATTEMPTS

public static final int METADATA_VERIFICATION_RETRY_ATTEMPTS

See Also:

Constant Field Values

METADATA_VERIFICATION_RETRY_BACKOFF_MULTIPLIER

public static final double METADATA_VERIFICATION_RETRY_BACKOFF_MULTIPLIER

See Also:

Constant Field Values

METADATA_VERIFICATION_RETRY_INITIAL_INTERVAL

public static final int METADATA_VERIFICATION_RETRY_INITIAL_INTERVAL

See Also:

Constant Field Values

METADATA_VERIFICATION_MAX_INTERVAL

public static final int METADATA_VERIFICATION_MAX_INTERVAL

See Also:

Constant Field Values

FETCH_SIZE

public static final String FETCH_SIZE

See Also:

Constant Field Values

QUEUE_SIZE

public static final String QUEUE_SIZE

See Also:

Constant Field Values

REQUIRED_ACKS

public static final String REQUIRED_ACKS

See Also:

Constant Field Values

COMPRESSION_CODEC

public static final String COMPRESSION_CODEC

See Also:

Constant Field Values

AUTO_COMMIT_ENABLED

public static final String AUTO_COMMIT_ENABLED

See Also:

Constant Field Values

utf8Serializer

public static final org.I0Itec.zkclient.serialize.ZkSerializer utf8Serializer

Used when writing directly to ZK. This is what Kafka expects.

PRODUCER_COMPRESSION_PROPERTIES

protected static final Set<Object> PRODUCER_COMPRESSION_PROPERTIES

Constructor Detail

KafkaMessageChannelBinder

public KafkaMessageChannelBinder(org.springframework.integration.kafka.support.ZookeeperConnect zookeeperConnect,
                                 String brokers,
                                 String zkAddress,
                                 String... headersToMap)

Method Detail

setOffsetStoreTopic

public void setOffsetStoreTopic(String offsetStoreTopic)

setOffsetStoreSegmentSize

public void setOffsetStoreSegmentSize(int offsetStoreSegmentSize)

setOffsetStoreRetentionTime

public void setOffsetStoreRetentionTime(int offsetStoreRetentionTime)

setSocketBufferSize

public void setSocketBufferSize(int socketBufferSize)

setOffsetStoreRequiredAcks

public void setOffsetStoreRequiredAcks(int offsetStoreRequiredAcks)

setOffsetStoreMaxFetchSize

public void setOffsetStoreMaxFetchSize(int offsetStoreMaxFetchSize)

setOffsetUpdateTimeWindow

public void setOffsetUpdateTimeWindow(int offsetUpdateTimeWindow)

setOffsetUpdateCount

public void setOffsetUpdateCount(int offsetUpdateCount)

setOffsetUpdateShutdownTimeout

public void setOffsetUpdateShutdownTimeout(int offsetUpdateShutdownTimeout)

setOffsetStoreBatchBytes

public void setOffsetStoreBatchBytes(int offsetStoreBatchBytes)

setOffsetStoreBatchTime

public void setOffsetStoreBatchTime(int offsetStoreBatchTime)

getConnectionFactory

public org.springframework.integration.kafka.core.ConnectionFactory getConnectionFactory()

setRetryOperations

public void setRetryOperations(org.springframework.retry.RetryOperations retryOperations)

Retry configuration for operations such as validating topic creation

Parameters:

retryOperations - the retry configuration

afterPropertiesSet

public void afterPropertiesSet()
                        throws Exception

Specified by:

afterPropertiesSet in interface org.springframework.beans.factory.InitializingBean

Overrides:

afterPropertiesSet in class MessageChannelBinderSupport

Throws:

Exception

validateTopicName

public static void validateTopicName(String topicName)

Allowed chars are ASCII alphanumerics, '.', '_' and '-'.

setDefaultReplicationFactor

public void setDefaultReplicationFactor(int defaultReplicationFactor)

setDefaultCompressionCodec

public void setDefaultCompressionCodec(String defaultCompressionCodec)

setDefaultRequiredAcks

public void setDefaultRequiredAcks(int defaultRequiredAcks)

setDefaultAutoCommitEnabled

public void setDefaultAutoCommitEnabled(boolean defaultAutoCommitEnabled)

Set the default auto commit enabled property; This is used to commit the offset either automatically or manually.

Parameters:

defaultAutoCommitEnabled -

setDefaultQueueSize

public void setDefaultQueueSize(int defaultQueueSize)

setDefaultFetchSize

public void setDefaultFetchSize(int defaultFetchSize)

setDefaultMinPartitionCount

public void setDefaultMinPartitionCount(int defaultMinPartitionCount)

setDefaultMaxWait

public void setDefaultMaxWait(int defaultMaxWait)

setMode

public void setMode(KafkaMessageChannelBinder.Mode mode)

isResetOffsets

public boolean isResetOffsets()

setResetOffsets

public void setResetOffsets(boolean resetOffsets)

getStartOffset

public KafkaMessageChannelBinder.StartOffset getStartOffset()

setStartOffset

public void setStartOffset(KafkaMessageChannelBinder.StartOffset startOffset)

doBindConsumer

protected Binding<org.springframework.messaging.MessageChannel> doBindConsumer(String name,
                                                                               String group,
                                                                               org.springframework.messaging.MessageChannel inputChannel,
                                                                               Properties properties)

Specified by:

doBindConsumer in class MessageChannelBinderSupport

bindProducer

public Binding<org.springframework.messaging.MessageChannel> bindProducer(String name,
                                                                          org.springframework.messaging.MessageChannel moduleOutputChannel,
                                                                          Properties properties)

createMessageListenerContainer

public org.springframework.integration.kafka.listener.KafkaMessageListenerContainer createMessageListenerContainer(Properties properties,
                                                                                                                   String group,
                                                                                                                   int maxConcurrency,
                                                                                                                   String topic,
                                                                                                                   long referencePoint)

doManualAck

public void doManualAck(LinkedList<org.springframework.messaging.MessageHeaders> messageHeadersList)

Overrides:

doManualAck in class MessageChannelBinderSupport