PeriodicalHyperlinkCollector

All Implemented Interfaces:

ai.platon.pulsar.common.collect.CrawlableFatLinkCollector , ai.platon.pulsar.common.collect.collector.DataCollector , ai.platon.pulsar.common.collect.collector.PriorityDataCollector , kotlin.Comparable
```
public class PeriodicalHyperlinkCollector
extends CircularHyperlinkCollector
                    
```

Nested Class Summary

Nested Classes
Modifier and Type	Class	Description
`public class`	`PeriodicalHyperlinkCollector.Companion`

Field Summary

Fields
Modifier and Type	Field	Description
`private String`	`name`
`private final Integer`	`size`
`private final Integer`	`estimatedSize`
`private final NormUrl`	`seed`
`private UrlNormalizerPipeline`	`urlNormalizer`
`private final ConcurrentSkipListMap<String, CrawlableFatLink>`	`fatLinks`
`private final PulsarSession`	`session`
`private final Queue<NormUrl>`	`seeds`
`private final Integer`	`capacity`
`private Integer`	`collectCount`
`private final Duration`	`collectTime`
`private Integer`	`collectedCount`
`private String`	`country`
`private final Instant`	`createTime`
`private Instant`	`deadTime`
`private String`	`district`
`private final Integer`	`estimatedExternalSize`
`private final Integer`	`externalSize`
`private Instant`	`firstCollectTime`
`private final Integer`	`id`
`private final Boolean`	`isDead`
`private final Set<String>`	`labels`
`private String`	`lang`
`private Instant`	`lastCollectedTime`
`private final Integer`	`priority`

Constructor Summary

Constructors
Constructor	Description
`PeriodicalHyperlinkCollector(PulsarSession session, NormUrl seed, Priority13 priority)`

Method Summary

Modifier and Type	Method	Description
`String`	`getName()`
`Unit`	`setName(String name)`
`Integer`	`getSize()`
`Integer`	`getEstimatedSize()`
`final NormUrl`	`getSeed()`
`final UrlNormalizerPipeline`	`getUrlNormalizer()`
`final Unit`	`setUrlNormalizer(UrlNormalizerPipeline urlNormalizer)`
`ConcurrentSkipListMap<String, CrawlableFatLink>`	`getFatLinks()`	Track the status of this batch, we need a notice when the batch is finished
`final PulsarSession`	`getSession()`	The pulsar session to use
`final Queue<NormUrl>`	`getSeeds()`	The urls of portal pages from where hyper links are extracted from
`Integer`	`getCapacity()`
`Integer`	`getCollectCount()`
`Unit`	`setCollectCount(Integer collectCount)`
`Duration`	`getCollectTime()`
`Integer`	`getCollectedCount()`
`Unit`	`setCollectedCount(Integer collectedCount)`
`String`	`getCountry()`
`Unit`	`setCountry(String country)`
`Instant`	`getCreateTime()`
`Instant`	`getDeadTime()`
`Unit`	`setDeadTime(Instant deadTime)`
`String`	`getDistrict()`
`Unit`	`setDistrict(String district)`
`Integer`	`getEstimatedExternalSize()`
`Integer`	`getExternalSize()`
`Instant`	`getFirstCollectTime()`
`Unit`	`setFirstCollectTime(Instant firstCollectTime)`
`Integer`	`getId()`
`Boolean`	`getIsDead()`
`Set<String>`	`getLabels()`
`String`	`getLang()`
`Unit`	`setLang(String lang)`
`Instant`	`getLastCollectedTime()`
`Unit`	`setLastCollectedTime(Instant lastCollectedTime)`
`Integer`	`getPriority()`
`Boolean`	`hasMore()`
`Integer`	`collectTo(List<UrlAware> sink)`

Methods inherited from class ai.platon.pulsar.common.collect.collector.AbstractPriorityDataCollector
collectTo, collectTo, collectTo, compareTo
Methods inherited from class ai.platon.pulsar.common.collect.PeriodicalHyperlinkCollector
clear
Methods inherited from class ai.platon.pulsar.common.collect.CircularHyperlinkCollector
dump, remove, remove
Methods inherited from class ai.platon.pulsar.common.collect.collector.AbstractDataCollector
deepClear
Methods inherited from class ai.platon.pulsar.common.collect.HyperlinkCollector
removeAll, removeAll, toString
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PeriodicalHyperlinkCollector

PeriodicalHyperlinkCollector(PulsarSession session, NormUrl seed, Priority13 priority)

Method Detail

getName
```
 String getName()
```

setName
```
 Unit setName(String name)
```

getSize
```
 Integer getSize()
```

getEstimatedSize
```
 Integer getEstimatedSize()
```

getSeed
```
 final NormUrl getSeed()
```

getUrlNormalizer

 final UrlNormalizerPipeline getUrlNormalizer()

setUrlNormalizer

 final Unit setUrlNormalizer(UrlNormalizerPipeline urlNormalizer)

getFatLinks
```
 ConcurrentSkipListMap<String, CrawlableFatLink> getFatLinks()
```
Track the status of this batch, we need a notice when the batch is finished

getSession
```
 final PulsarSession getSession()
```
The pulsar session to use

getSeeds
```
 final Queue<NormUrl> getSeeds()
```
The urls of portal pages from where hyper links are extracted from

getCapacity
```
 Integer getCapacity()
```

getCollectCount
```
 Integer getCollectCount()
```

setCollectCount

 Unit setCollectCount(Integer collectCount)

getCollectTime
```
 Duration getCollectTime()
```

getCollectedCount
```
 Integer getCollectedCount()
```

setCollectedCount

 Unit setCollectedCount(Integer collectedCount)

getCountry
```
 String getCountry()
```

setCountry
```
 Unit setCountry(String country)
```

getCreateTime
```
 Instant getCreateTime()
```

getDeadTime
```
 Instant getDeadTime()
```

setDeadTime
```
 Unit setDeadTime(Instant deadTime)
```

getDistrict
```
 String getDistrict()
```

setDistrict
```
 Unit setDistrict(String district)
```

getEstimatedExternalSize
```
 Integer getEstimatedExternalSize()
```

getExternalSize
```
 Integer getExternalSize()
```

getFirstCollectTime
```
 Instant getFirstCollectTime()
```

setFirstCollectTime

 Unit setFirstCollectTime(Instant firstCollectTime)

getId
```
 Integer getId()
```

getIsDead
```
 Boolean getIsDead()
```

getLabels
```
 Set<String> getLabels()
```

getLang
```
 String getLang()
```

setLang
```
 Unit setLang(String lang)
```

getLastCollectedTime
```
 Instant getLastCollectedTime()
```

setLastCollectedTime

 Unit setLastCollectedTime(Instant lastCollectedTime)

getPriority
```
 Integer getPriority()
```

hasMore
```
 Boolean hasMore()
```

collectTo

 Integer collectTo(List<UrlAware> sink)

Class PeriodicalHyperlinkCollector

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class ai.platon.pulsar.common.collect.collector.AbstractPriorityDataCollector

Methods inherited from class ai.platon.pulsar.common.collect.PeriodicalHyperlinkCollector

Methods inherited from class ai.platon.pulsar.common.collect.CircularHyperlinkCollector

Methods inherited from class ai.platon.pulsar.common.collect.collector.AbstractDataCollector

Methods inherited from class ai.platon.pulsar.common.collect.HyperlinkCollector

Methods inherited from class java.lang.Object

Constructor Detail

PeriodicalHyperlinkCollector

Method Detail

getName

setName

getSize

getEstimatedSize

getSeed

getUrlNormalizer

setUrlNormalizer

getFatLinks

getSession

getSeeds

getCapacity

getCollectCount

setCollectCount

getCollectTime

getCollectedCount

setCollectedCount

getCountry

setCountry

getCreateTime

getDeadTime

setDeadTime

getDistrict

setDistrict

getEstimatedExternalSize

getExternalSize

getFirstCollectTime

setFirstCollectTime

getId

getIsDead

getLabels

getLang

setLang

getLastCollectedTime

setLastCollectedTime

getPriority

hasMore

collectTo