org.apache.pulsar.broker.loadbalance.impl

Class LoadManagerShared

java.lang.Object

org.apache.pulsar.broker.loadbalance.impl.LoadManagerShared

public class LoadManagerShared
extends Object

This class contains code which in shared between the two load manager implementations.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	LoadManagerShared.BrokerTopicLoadingPredicate

Field Summary

Fields

Modifier and Type	Field and Description
static long	LOAD_REPORT_UPDATE_MIMIMUM_INTERVAL
static org.slf4j.Logger	log
static int	MIBI

Method Summary

Modifier and Type	Method and Description
static void	applyNamespacePolicies(org.apache.pulsar.common.naming.ServiceUnitId serviceUnit, SimpleResourceAllocationPolicies policies, Set<String> brokerCandidateCache, Set<String> availableBrokers, LoadManagerShared.BrokerTopicLoadingPredicate brokerTopicLoadingPredicate)
static void	fillNamespaceToBundlesMap(Set<String> bundles, org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet<String>> target) Using the given bundles, populate the namespace to bundle range map.
static void	filterAntiAffinityGroupOwnedBrokers(PulsarService pulsar, String assignedBundleName, Set<String> candidates, org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet<String>>> brokerToNamespaceToBundleRange, Map<String,String> brokerToDomainMap) It tries to filter out brokers which own namespace with same anti-affinity-group as given namespace.
static void	filterBrokersWithLargeTopicCount(Set<String> brokerCandidateCache, LoadData loadData, int loadBalancerBrokerMaxTopics) It filters out brokers which owns topic higher than configured threshold at ServiceConfiguration.loadBalancerBrokerMaxTopics.
static CompletableFuture<Map<String,Integer>>	getAntiAffinityNamespaceOwnedBrokers(PulsarService pulsar, String namespaceName, org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet<String>>> brokerToNamespaceToBundleRange) It returns map of broker and count of namespace that are belong to the same anti-affinity group as given
static String	getBundleRangeFromBundleName(String bundleName)
static String	getNamespaceNameFromBundleName(String bundleName)
static org.apache.pulsar.policies.data.loadbalancer.SystemResourceUsage	getSystemResourceUsage(BrokerHostUsage brokerHostUsage)
static boolean	isLoadSheddingEnabled(PulsarService pulsar) If load balancing is enabled, load shedding is enabled by default unless forced off by dynamic configuration
static void	removeMostServicingBrokersForNamespace(String assignedBundleName, Set<String> candidates, org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet<String>>> brokerToNamespaceToBundleRange) Removes the brokers which have more bundles assigned to them in the same namespace as the incoming bundle than at least one other available broker from consideration.
static boolean	shouldAntiAffinityNamespaceUnload(String namespace, String bundle, String currentBroker, PulsarService pulsar, org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet<String>>> brokerToNamespaceToBundleRange, Set<String> candidateBrokers) It checks if given anti-affinity namespace should be unloaded by broker due to load-shedding.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

public static final org.slf4j.Logger log

MIBI

public static final int MIBI

See Also:

Constant Field Values

LOAD_REPORT_UPDATE_MIMIMUM_INTERVAL

public static final long LOAD_REPORT_UPDATE_MIMIMUM_INTERVAL

Method Detail

applyNamespacePolicies

public static void applyNamespacePolicies(org.apache.pulsar.common.naming.ServiceUnitId serviceUnit,
                                          SimpleResourceAllocationPolicies policies,
                                          Set<String> brokerCandidateCache,
                                          Set<String> availableBrokers,
                                          LoadManagerShared.BrokerTopicLoadingPredicate brokerTopicLoadingPredicate)

fillNamespaceToBundlesMap

public static void fillNamespaceToBundlesMap(Set<String> bundles,
                                             org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet<String>> target)

Using the given bundles, populate the namespace to bundle range map.

Parameters:

bundles - Bundles with which to populate.

target - Map to fill.

getBundleRangeFromBundleName

public static String getBundleRangeFromBundleName(String bundleName)

getNamespaceNameFromBundleName

public static String getNamespaceNameFromBundleName(String bundleName)

getSystemResourceUsage

public static org.apache.pulsar.policies.data.loadbalancer.SystemResourceUsage getSystemResourceUsage(BrokerHostUsage brokerHostUsage)
                                                                                               throws IOException

Throws:

IOException

isLoadSheddingEnabled

public static boolean isLoadSheddingEnabled(PulsarService pulsar)

If load balancing is enabled, load shedding is enabled by default unless forced off by dynamic configuration

Returns:

true by default

removeMostServicingBrokersForNamespace

public static void removeMostServicingBrokersForNamespace(String assignedBundleName,
                                                          Set<String> candidates,
                                                          org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet<String>>> brokerToNamespaceToBundleRange)

Removes the brokers which have more bundles assigned to them in the same namespace as the incoming bundle than at least one other available broker from consideration.

Parameters:

assignedBundleName - Name of bundle to be assigned.

candidates - BrokersBase available for placement.

brokerToNamespaceToBundleRange - Map from brokers to namespaces to bundle ranges.

filterAntiAffinityGroupOwnedBrokers

public static void filterAntiAffinityGroupOwnedBrokers(PulsarService pulsar,
                                                       String assignedBundleName,
                                                       Set<String> candidates,
                                                       org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet<String>>> brokerToNamespaceToBundleRange,
                                                       Map<String,String> brokerToDomainMap)

It tries to filter out brokers which own namespace with same anti-affinity-group as given namespace. If all the domains own namespace with same anti-affinity group then it will try to keep brokers with domain that has least number of namespaces. It also tries to keep brokers which has least number of namespace with in domain. eg.

 Before:
 Domain-count  BrokersBase-count
 ____________  ____________
 d1-3          b1-2,b2-1
 d2-3          b3-2,b4-1
 d3-4          b5-2,b6-2

 After filtering: "candidates" brokers
 Domain-count  BrokersBase-count
 ____________  ____________
 d1-3          b2-1
 d2-3          b4-1

 "candidate" broker to own anti-affinity-namespace = b2 or b4

 

Parameters:

pulsar -

assignedBundleName -

candidates -

brokerToNamespaceToBundleRange -

getAntiAffinityNamespaceOwnedBrokers

public static CompletableFuture<Map<String,Integer>> getAntiAffinityNamespaceOwnedBrokers(PulsarService pulsar,
                                                                                          String namespaceName,
                                                                                          org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet<String>>> brokerToNamespaceToBundleRange)

It returns map of broker and count of namespace that are belong to the same anti-affinity group as given

Parameters:

pulsar -

namespaceName -

brokerToNamespaceToBundleRange -

Returns:

shouldAntiAffinityNamespaceUnload

public static boolean shouldAntiAffinityNamespaceUnload(String namespace,
                                                        String bundle,
                                                        String currentBroker,
                                                        PulsarService pulsar,
                                                        org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap<String,org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet<String>>> brokerToNamespaceToBundleRange,
                                                        Set<String> candidateBrokers)
                                                 throws Exception

It checks if given anti-affinity namespace should be unloaded by broker due to load-shedding. If all the brokers are owning same number of anti-affinity namespaces then unloading this namespace again ends up at the same broker from which it was unloaded. So, this util checks that given namespace should be unloaded only if it can be loaded by different broker.

Parameters:

namespace -

bundle -

currentBroker -

pulsar -

brokerToNamespaceToBundleRange -

candidateBrokers -

Returns:

Throws:

Exception

filterBrokersWithLargeTopicCount

public static void filterBrokersWithLargeTopicCount(Set<String> brokerCandidateCache,
                                                    LoadData loadData,
                                                    int loadBalancerBrokerMaxTopics)

It filters out brokers which owns topic higher than configured threshold at ServiceConfiguration.loadBalancerBrokerMaxTopics.
if all the brokers own topic higher than threshold then it resets the list with original broker candidates

Parameters:

brokerCandidateCache -

loadData -

loadBalancerBrokerMaxTopics -