org.apache.pulsar.broker.service.persistent

Class PersistentTopic

java.lang.Object

org.apache.pulsar.broker.service.AbstractTopic

org.apache.pulsar.broker.service.persistent.PersistentTopic

All Implemented Interfaces:

AsyncCallbacks.AddEntryCallback, Topic, TopicPolicyListener<org.apache.pulsar.common.policies.data.TopicPolicies>

Direct Known Subclasses:

SystemTopic

public class PersistentTopic
extends AbstractTopic
implements Topic, AsyncCallbacks.AddEntryCallback, TopicPolicyListener<org.apache.pulsar.common.policies.data.TopicPolicies>

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.pulsar.broker.service.Topic

Topic.PublishContext

Field Summary

Fields

Modifier and Type	Field and Description
boolean	delayedDeliveryEnabled
long	delayedDeliveryTickTimeMillis
protected ManagedLedger	ledger
static int	MESSAGE_RATE_BACKOFF_MS
protected MessageDeduplication	messageDeduplication
boolean	msgChunkPublished
protected static AtomicLongFieldUpdater<PersistentTopic>	USAGE_COUNT_UPDATER

Fields inherited from class org.apache.pulsar.broker.service.AbstractTopic

addEntryLatencyStatsUsec, brokerService, hasBatchMessagePublished, inactiveTopicPolicies, isAllowAutoUpdateSchema, isEncryptionRequired, isFenced, lastActive, lock, maxSubscriptionsPerTopic, maxUnackedMessagesOnConsumer, preciseTopicPublishRateLimitingEnable, producers, replicatorPrefix, schemaCompatibilityStrategy, schemaValidationEnforced, topic, topicPublishRateLimiter, transactionBuffer, transactionBufferLock

Constructor Summary

Constructors

Constructor and Description
PersistentTopic(String topic, ManagedLedger ledger, BrokerService brokerService)

Method Summary

Modifier and Type	Method and Description
void	addComplete(Position pos, Object ctx)
void	addFailed(ManagedLedgerException exception, Object ctx)
void	addProducer(Producer producer)
protected boolean	addReplicationCluster(String remoteCluster, PersistentTopic persistentTopic, String replicatorName, String localCluster)
CompletableFuture<Void>	addSchemaIfIdleOrCheckCompatible(org.apache.pulsar.common.protocol.schema.SchemaData schema) If the topic is idle (no producers, no entries, no subscribers and no existing schema), add the passed schema to the topic.
void	asyncReadEntry(PositionImpl position, AsyncCallbacks.ReadEntryCallback callback, Object ctx)
void	checkBackloggedCursors() Activate cursors those caught up backlog-threshold entries and deactivate slow cursors which are creating backlog.
void	checkCompaction()
void	checkDeduplicationSnapshot()
CompletableFuture<Void>	checkDeduplicationStatus()
void	checkGC()
void	checkInactiveSubscriptions()
void	checkMessageDeduplicationInfo()
void	checkMessageExpiry()
CompletableFuture<Void>	checkReplication()
CompletableFuture<Void>	clearBacklog() Clears backlog for all cursors in the topic
CompletableFuture<Void>	clearBacklog(String cursorName) Clears backlog for a given cursor in the topic.
CompletableFuture<Void>	close()
CompletableFuture<Void>	close(boolean closeWithoutWaitingClientDisconnect) Close this topic - close all producers and subscriptions associated with this topic
LongRunningProcessStatus	compactionStatus()
CompletableFuture<Subscription>	createSubscription(String subscriptionName, org.apache.pulsar.common.api.proto.PulsarApi.CommandSubscribe.InitialPosition initialPosition, boolean replicateSubscriptionState)
CompletableFuture<Void>	delete() Delete the managed ledger associated with this topic
CompletableFuture<Void>	deleteForcefully() Forcefully close all producers/consumers/replicators and deletes the topic.
CompletableFuture<Void>	endTxn(org.apache.pulsar.client.api.transaction.TxnID txnID, int txnAction, List<org.apache.pulsar.common.api.proto.PulsarApi.MessageIdData> sendMessageList) End the transaction in this topic.
org.apache.pulsar.common.policies.data.BacklogQuota	getBacklogQuota()
long	getBacklogSize()
CompactedTopic	getCompactedTopic()
long	getDelayedDeliveryTickTimeMillis()
Optional<DispatchRateLimiter>	getDispatchRateLimiter()
PositionImpl	getFirstPosition()
CompletableFuture<org.apache.pulsar.common.policies.data.PersistentTopicInternalStats>	getInternalStats(boolean includeLedgerMetadata)
CompletableFuture<org.apache.pulsar.client.api.MessageId>	getLastMessageId()
Position	getLastPosition()
long	getLastPublishedSequenceId(String producerName)
double	getLastUpdatedAvgPublishRateInByte()
double	getLastUpdatedAvgPublishRateInMsg()
ManagedLedger	getManagedLedger()
int	getMaxUnackedMessagesOnConsumer()
int	getMaxUnackedMessagesOnSubscription()
MessageDeduplication	getMessageDeduplication()
int	getNumberOfConsumers()
long	getNumberOfEntries()
Replicator	getPersistentReplicator(String remoteCluster)
PositionImpl	getPositionAfterN(PositionImpl startPosition, long n)
org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,Replicator>	getReplicators()
org.apache.pulsar.common.policies.data.TopicStats	getStats(boolean getPreciseBacklog, boolean subscriptionBacklogSize)
Optional<SubscribeRateLimiter>	getSubscribeRateLimiter()
PersistentSubscription	getSubscription(String subscriptionName)
org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,PersistentSubscription>	getSubscriptions()
CompletableFuture<TransactionBuffer>	getTransactionBuffer(boolean createIfMissing) Get the $TransactionBuffer of this Topic.
protected void	handleProducerRemoved(Producer producer)
boolean	isActive(org.apache.pulsar.common.policies.data.InactiveTopicDeleteMode deleteMode)
boolean	isBacklogExceeded()
boolean	isBacklogQuotaExceeded(String producerName)
boolean	isDeduplicationEnabled()
boolean	isDelayedDeliveryEnabled()
boolean	isOldestMessageExpired(ManagedCursor cursor, long messageTTLInSeconds)
boolean	isReplicated()
boolean	isSystemTopic()
OffloadProcessStatus	offloadStatus()
CompletableFuture<Void>	onPoliciesUpdate(org.apache.pulsar.common.policies.data.Policies data)
void	onUpdate(org.apache.pulsar.common.policies.data.TopicPolicies policies)
void	publishMessage(io.netty.buffer.ByteBuf headersAndPayload, Topic.PublishContext publishContext)
void	publishTxnMessage(org.apache.pulsar.client.api.transaction.TxnID txnID, io.netty.buffer.ByteBuf headersAndPayload, Topic.PublishContext publishContext) Publish Transaction message to this Topic's TransactionBuffer
void	removeProducer(Producer producer)
void	startReplProducers()
CompletableFuture<Void>	stopReplProducers()
CompletableFuture<Consumer>	subscribe(TransportCnx cnx, String subscriptionName, long consumerId, org.apache.pulsar.common.api.proto.PulsarApi.CommandSubscribe.SubType subType, int priorityLevel, String consumerName, boolean isDurable, org.apache.pulsar.client.api.MessageId startMessageId, Map<String,String> metadata, boolean readCompacted, org.apache.pulsar.common.api.proto.PulsarApi.CommandSubscribe.InitialPosition initialPosition, long startMessageRollbackDurationSec, boolean replicatedSubscriptionState, org.apache.pulsar.common.api.proto.PulsarApi.KeySharedMeta keySharedMeta)
CompletableFuture<org.apache.pulsar.client.api.MessageId>	terminate()
void	triggerCompaction()
void	triggerOffload(MessageIdImpl messageId)
CompletableFuture<Void>	unsubscribe(String subscriptionName) Delete the cursor ledger for a given subscription
void	updateRates(NamespaceStats nsStats, org.apache.pulsar.policies.data.loadbalancer.NamespaceBundleStats bundleStats, StatsOutputStream topicStatsStream, ClusterReplicationMetrics replStats, String namespace, boolean hydratePublishers)

Methods inherited from class org.apache.pulsar.broker.service.AbstractTopic

addConsumerToSubscription, addSchema, checkSchemaCompatibleForConsumer, checkTopicFenced, checkTopicPublishThrottlingRate, deletePartitionedTopicMetadataWhileInactive, deleteSchema, disableCnxAutoRead, disableProducerRead, enableCnxAutoRead, enableProducerReadForPublishBufferLimiting, enableProducerReadForPublishRateLimiting, getBrokerPublishRateLimiter, getBrokerService, getBytesInCounter, getBytesOutCounter, getInactiveTopicPolicies, getMsgInCounter, getMsgOutCounter, getName, getProducers, getReplicatorPrefix, getSchemaValidationEnforced, getTopicPolicies, getTopicPublishRateLimiter, hasLocalProducers, hasSchema, incrementPublishCount, internalAddProducer, isBrokerPublishRateExceeded, isConsumersExceededOnTopic, isDeleteWhileInactive, isEncryptionRequired, isExceedMaximumMessageSize, isProducersExceeded, isPublishRateExceeded, isTopicPublishRateExceeded, markBatchMessagePublished, recordAddLatency, resetBrokerPublishCountAndEnableReadIfRequired, resetInactiveTopicPolicies, resetTopicPublishCountAndEnableReadIfRequired, setDeleteWhileInactive, setSchemaCompatibilityStrategy, toString, updateMaxPublishRate, updatePublishDispatcher

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.pulsar.broker.service.Topic

addSchema, checkSchemaCompatibleForConsumer, checkTopicPublishThrottlingRate, deleteSchema, disableCnxAutoRead, enableCnxAutoRead, getName, getProducers, getSchemaValidationEnforced, hasSchema, incrementPublishCount, isBrokerPublishRateExceeded, isEncryptionRequired, isPublishRateExceeded, isTopicPublishRateExceeded, recordAddLatency, resetBrokerPublishCountAndEnableReadIfRequired, resetTopicPublishCountAndEnableReadIfRequired

Field Detail

ledger

protected final ManagedLedger ledger

USAGE_COUNT_UPDATER

protected static final AtomicLongFieldUpdater<PersistentTopic> USAGE_COUNT_UPDATER

msgChunkPublished

public boolean msgChunkPublished

delayedDeliveryTickTimeMillis

public volatile long delayedDeliveryTickTimeMillis

delayedDeliveryEnabled

public volatile boolean delayedDeliveryEnabled

MESSAGE_RATE_BACKOFF_MS

public static final int MESSAGE_RATE_BACKOFF_MS

See Also:

Constant Field Values

messageDeduplication

protected final MessageDeduplication messageDeduplication

Constructor Detail

PersistentTopic

public PersistentTopic(String topic,
                       ManagedLedger ledger,
                       BrokerService brokerService)
                throws BrokerServiceException.NamingException

Throws:

BrokerServiceException.NamingException

Method Detail

publishMessage

public void publishMessage(io.netty.buffer.ByteBuf headersAndPayload,
                           Topic.PublishContext publishContext)

Specified by:

publishMessage in interface Topic

asyncReadEntry

public void asyncReadEntry(PositionImpl position,
                           AsyncCallbacks.ReadEntryCallback callback,
                           Object ctx)

getPositionAfterN

public PositionImpl getPositionAfterN(PositionImpl startPosition,
                                      long n)
                               throws ManagedLedgerException

Throws:

ManagedLedgerException

getFirstPosition

public PositionImpl getFirstPosition()
                              throws ManagedLedgerException

Throws:

ManagedLedgerException

getNumberOfEntries

public long getNumberOfEntries()

addComplete

public void addComplete(Position pos,
                        Object ctx)

Specified by:

addComplete in interface AsyncCallbacks.AddEntryCallback

addFailed

public void addFailed(ManagedLedgerException exception,
                      Object ctx)

Specified by:

addFailed in interface AsyncCallbacks.AddEntryCallback

addProducer

public void addProducer(Producer producer)
                 throws BrokerServiceException

Specified by:

addProducer in interface Topic

Throws:

BrokerServiceException

startReplProducers

public void startReplProducers()

stopReplProducers

public CompletableFuture<Void> stopReplProducers()

removeProducer

public void removeProducer(Producer producer)

Specified by:

removeProducer in interface Topic

handleProducerRemoved

protected void handleProducerRemoved(Producer producer)

Specified by:

handleProducerRemoved in class AbstractTopic

subscribe

public CompletableFuture<Consumer> subscribe(TransportCnx cnx,
                                             String subscriptionName,
                                             long consumerId,
                                             org.apache.pulsar.common.api.proto.PulsarApi.CommandSubscribe.SubType subType,
                                             int priorityLevel,
                                             String consumerName,
                                             boolean isDurable,
                                             org.apache.pulsar.client.api.MessageId startMessageId,
                                             Map<String,String> metadata,
                                             boolean readCompacted,
                                             org.apache.pulsar.common.api.proto.PulsarApi.CommandSubscribe.InitialPosition initialPosition,
                                             long startMessageRollbackDurationSec,
                                             boolean replicatedSubscriptionState,
                                             org.apache.pulsar.common.api.proto.PulsarApi.KeySharedMeta keySharedMeta)

Specified by:

subscribe in interface Topic

createSubscription

public CompletableFuture<Subscription> createSubscription(String subscriptionName,
                                                          org.apache.pulsar.common.api.proto.PulsarApi.CommandSubscribe.InitialPosition initialPosition,
                                                          boolean replicateSubscriptionState)

Specified by:

createSubscription in interface Topic

unsubscribe

public CompletableFuture<Void> unsubscribe(String subscriptionName)

Delete the cursor ledger for a given subscription

Specified by:

unsubscribe in interface Topic

Parameters:

subscriptionName - Subscription for which the cursor ledger is to be deleted

Returns:

Completable future indicating completion of unsubscribe operation Completed exceptionally with: ManagedLedgerException if cursor ledger delete fails

delete

public CompletableFuture<Void> delete()

Delete the managed ledger associated with this topic

Specified by:

delete in interface Topic

Returns:

Completable future indicating completion of delete operation Completed exceptionally with: IllegalStateException if topic is still active ManagedLedgerException if ledger delete operation fails

deleteForcefully

public CompletableFuture<Void> deleteForcefully()

Forcefully close all producers/consumers/replicators and deletes the topic. this function is used when local cluster is removed from global-namespace replication list. Because broker doesn't allow lookup if local cluster is not part of replication cluster list.

Specified by:

deleteForcefully in interface Topic

Returns:

close

public CompletableFuture<Void> close()

close

public CompletableFuture<Void> close(boolean closeWithoutWaitingClientDisconnect)

Close this topic - close all producers and subscriptions associated with this topic

Specified by:

close in interface Topic

Parameters:

closeWithoutWaitingClientDisconnect - don't wait for client disconnect and forcefully close managed-ledger

Returns:

Completable future indicating completion of close operation

checkDeduplicationStatus

public CompletableFuture<Void> checkDeduplicationStatus()

checkReplication

public CompletableFuture<Void> checkReplication()

Specified by:

checkReplication in interface Topic

checkMessageExpiry

public void checkMessageExpiry()

Specified by:

checkMessageExpiry in interface Topic

checkMessageDeduplicationInfo

public void checkMessageDeduplicationInfo()

Specified by:

checkMessageDeduplicationInfo in interface Topic

checkCompaction

public void checkCompaction()

addReplicationCluster

protected boolean addReplicationCluster(String remoteCluster,
                                        PersistentTopic persistentTopic,
                                        String replicatorName,
                                        String localCluster)

isDeduplicationEnabled

public boolean isDeduplicationEnabled()

getNumberOfConsumers

public int getNumberOfConsumers()

Specified by:

getNumberOfConsumers in class AbstractTopic

getSubscriptions

public org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,PersistentSubscription> getSubscriptions()

Specified by:

getSubscriptions in interface Topic

getSubscription

public PersistentSubscription getSubscription(String subscriptionName)

Specified by:

getSubscription in interface Topic

getReplicators

public org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,Replicator> getReplicators()

Specified by:

getReplicators in interface Topic

getPersistentReplicator

public Replicator getPersistentReplicator(String remoteCluster)

getManagedLedger

public ManagedLedger getManagedLedger()

updateRates

public void updateRates(NamespaceStats nsStats,
                        org.apache.pulsar.policies.data.loadbalancer.NamespaceBundleStats bundleStats,
                        StatsOutputStream topicStatsStream,
                        ClusterReplicationMetrics replStats,
                        String namespace,
                        boolean hydratePublishers)

Specified by:

updateRates in interface Topic

getLastUpdatedAvgPublishRateInMsg

public double getLastUpdatedAvgPublishRateInMsg()

getLastUpdatedAvgPublishRateInByte

public double getLastUpdatedAvgPublishRateInByte()

getStats

public org.apache.pulsar.common.policies.data.TopicStats getStats(boolean getPreciseBacklog,
                                                                  boolean subscriptionBacklogSize)

Specified by:

getStats in interface Topic

getInternalStats

public CompletableFuture<org.apache.pulsar.common.policies.data.PersistentTopicInternalStats> getInternalStats(boolean includeLedgerMetadata)

Specified by:

getInternalStats in interface Topic

getBacklogSize

public long getBacklogSize()

isActive

public boolean isActive(org.apache.pulsar.common.policies.data.InactiveTopicDeleteMode deleteMode)

checkGC

public void checkGC()

Specified by:

checkGC in interface Topic

checkInactiveSubscriptions

public void checkInactiveSubscriptions()

Specified by:

checkInactiveSubscriptions in interface Topic

checkBackloggedCursors

public void checkBackloggedCursors()

Description copied from interface: Topic

Activate cursors those caught up backlog-threshold entries and deactivate slow cursors which are creating backlog.

Specified by:

checkBackloggedCursors in interface Topic

checkDeduplicationSnapshot

public void checkDeduplicationSnapshot()

Specified by:

checkDeduplicationSnapshot in interface Topic

onPoliciesUpdate

public CompletableFuture<Void> onPoliciesUpdate(org.apache.pulsar.common.policies.data.Policies data)

Specified by:

onPoliciesUpdate in interface Topic

getBacklogQuota

public org.apache.pulsar.common.policies.data.BacklogQuota getBacklogQuota()

Specified by:

getBacklogQuota in interface Topic

Returns:

Backlog quota for topic

isBacklogQuotaExceeded

public boolean isBacklogQuotaExceeded(String producerName)

Specified by:

isBacklogQuotaExceeded in interface Topic

Returns:

quota exceeded status for blocking producer creation

isBacklogExceeded

public boolean isBacklogExceeded()

Returns:

determine if quota enforcement needs to be done for topic

isReplicated

public boolean isReplicated()

Specified by:

isReplicated in interface Topic

terminate

public CompletableFuture<org.apache.pulsar.client.api.MessageId> terminate()

isOldestMessageExpired

public boolean isOldestMessageExpired(ManagedCursor cursor,
                                      long messageTTLInSeconds)

clearBacklog

public CompletableFuture<Void> clearBacklog()

Clears backlog for all cursors in the topic

Returns:

clearBacklog

public CompletableFuture<Void> clearBacklog(String cursorName)

Clears backlog for a given cursor in the topic.

Note: For a replication cursor, just provide the remote cluster name

Parameters:

cursorName -

Returns:

getDispatchRateLimiter

public Optional<DispatchRateLimiter> getDispatchRateLimiter()

Specified by:

getDispatchRateLimiter in interface Topic

getSubscribeRateLimiter

public Optional<SubscribeRateLimiter> getSubscribeRateLimiter()

getLastPublishedSequenceId

public long getLastPublishedSequenceId(String producerName)

getLastPosition

public Position getLastPosition()

Specified by:

getLastPosition in interface Topic

getLastMessageId

public CompletableFuture<org.apache.pulsar.client.api.MessageId> getLastMessageId()

Specified by:

getLastMessageId in interface Topic

triggerCompaction

public void triggerCompaction()
                       throws PulsarServerException,
                              BrokerServiceException.AlreadyRunningException

Throws:

PulsarServerException

BrokerServiceException.AlreadyRunningException

compactionStatus

public LongRunningProcessStatus compactionStatus()

triggerOffload

public void triggerOffload(MessageIdImpl messageId)
                    throws BrokerServiceException.AlreadyRunningException

Throws:

BrokerServiceException.AlreadyRunningException

offloadStatus

public OffloadProcessStatus offloadStatus()

addSchemaIfIdleOrCheckCompatible

public CompletableFuture<Void> addSchemaIfIdleOrCheckCompatible(org.apache.pulsar.common.protocol.schema.SchemaData schema)

Description copied from interface: Topic

If the topic is idle (no producers, no entries, no subscribers and no existing schema), add the passed schema to the topic. Otherwise, check that the passed schema is compatible with what the topic already has.

Specified by:

addSchemaIfIdleOrCheckCompatible in interface Topic

getCompactedTopic

public CompactedTopic getCompactedTopic()

isSystemTopic

public boolean isSystemTopic()

Specified by:

isSystemTopic in interface Topic

getTransactionBuffer

public CompletableFuture<TransactionBuffer> getTransactionBuffer(boolean createIfMissing)

Description copied from interface: Topic

Get the $TransactionBuffer of this Topic.

Specified by:

getTransactionBuffer in interface Topic

Parameters:

createIfMissing - Create the TransactionBuffer if missing.

Returns:

TransactionBuffer CompletableFuture

publishTxnMessage

public void publishTxnMessage(org.apache.pulsar.client.api.transaction.TxnID txnID,
                              io.netty.buffer.ByteBuf headersAndPayload,
                              Topic.PublishContext publishContext)

Description copied from interface: Topic

Publish Transaction message to this Topic's TransactionBuffer

Specified by:

publishTxnMessage in interface Topic

Parameters:

txnID - Transaction Id

headersAndPayload - Message data

publishContext - Publish context

endTxn

public CompletableFuture<Void> endTxn(org.apache.pulsar.client.api.transaction.TxnID txnID,
                                      int txnAction,
                                      List<org.apache.pulsar.common.api.proto.PulsarApi.MessageIdData> sendMessageList)

Description copied from interface: Topic

End the transaction in this topic.

Specified by:

endTxn in interface Topic

Parameters:

txnID - Transaction id

txnAction - Transaction action.

Returns:

getDelayedDeliveryTickTimeMillis

public long getDelayedDeliveryTickTimeMillis()

getMaxUnackedMessagesOnConsumer

public int getMaxUnackedMessagesOnConsumer()

isDelayedDeliveryEnabled

public boolean isDelayedDeliveryEnabled()

getMaxUnackedMessagesOnSubscription

public int getMaxUnackedMessagesOnSubscription()

onUpdate

public void onUpdate(org.apache.pulsar.common.policies.data.TopicPolicies policies)

Specified by:

onUpdate in interface TopicPolicyListener<org.apache.pulsar.common.policies.data.TopicPolicies>

getMessageDeduplication

public MessageDeduplication getMessageDeduplication()