HyperlinkCollector

All Implemented Interfaces:

ai.platon.pulsar.common.collect.CrawlableFatLinkCollector , ai.platon.pulsar.common.collect.collector.DataCollector , ai.platon.pulsar.common.collect.collector.PriorityDataCollector , kotlin.Comparable
```
public class HyperlinkCollector
extends AbstractPriorityDataCollector<UrlAware> implements CrawlableFatLinkCollector
                    
```
Collect hyper links from the given seeds. The urls are restricted by loadArguments and urlNormalizer.
- all urls are restricted by css outLinkSelector
- all urls are restricted by urlPattern
- all urls have to not be fetched before or expired against the last version

Field Summary

Fields
Modifier and Type	Field	Description
`private UrlNormalizerPipeline`	`urlNormalizer`
`private String`	`name`
`private final Integer`	`size`
`private final Integer`	`estimatedSize`
`private final ConcurrentSkipListMap<String, CrawlableFatLink>`	`fatLinks`
`private final PulsarSession`	`session`
`private final Queue<NormUrl>`	`seeds`
`private final Integer`	`capacity`
`private Integer`	`collectCount`
`private final Duration`	`collectTime`
`private Integer`	`collectedCount`
`private String`	`country`
`private final Instant`	`createTime`
`private Instant`	`deadTime`
`private String`	`district`
`private final Integer`	`estimatedExternalSize`
`private final Integer`	`externalSize`
`private Instant`	`firstCollectTime`
`private final Integer`	`id`
`private final Boolean`	`isDead`
`private final Set<String>`	`labels`
`private String`	`lang`
`private Instant`	`lastCollectedTime`
`private final Integer`	`priority`

Constructor Summary

Constructors
Constructor	Description
`HyperlinkCollector(PulsarSession session, Queue<NormUrl> seeds, Priority13 priority)`

Method Summary

Modifier and Type	Method	Description
`final UrlNormalizerPipeline`	`getUrlNormalizer()`
`final Unit`	`setUrlNormalizer(UrlNormalizerPipeline urlNormalizer)`
`String`	`getName()`
`Unit`	`setName(String name)`
`Integer`	`getSize()`
`Integer`	`getEstimatedSize()`
`ConcurrentSkipListMap<String, CrawlableFatLink>`	`getFatLinks()`	Track the status of this batch, we need a notice when the batch is finished
`final PulsarSession`	`getSession()`	The pulsar session to use
`final Queue<NormUrl>`	`getSeeds()`	The urls of portal pages from where hyper links are extracted from
`Integer`	`getCapacity()`
`Integer`	`getCollectCount()`
`Unit`	`setCollectCount(Integer collectCount)`
`Duration`	`getCollectTime()`
`Integer`	`getCollectedCount()`
`Unit`	`setCollectedCount(Integer collectedCount)`
`String`	`getCountry()`
`Unit`	`setCountry(String country)`
`Instant`	`getCreateTime()`
`Instant`	`getDeadTime()`
`Unit`	`setDeadTime(Instant deadTime)`
`String`	`getDistrict()`
`Unit`	`setDistrict(String district)`
`Integer`	`getEstimatedExternalSize()`
`Integer`	`getExternalSize()`
`Instant`	`getFirstCollectTime()`
`Unit`	`setFirstCollectTime(Instant firstCollectTime)`
`Integer`	`getId()`
`Boolean`	`getIsDead()`
`Set<String>`	`getLabels()`
`String`	`getLang()`
`Unit`	`setLang(String lang)`
`Instant`	`getLastCollectedTime()`
`Unit`	`setLastCollectedTime(Instant lastCollectedTime)`
`Integer`	`getPriority()`
`CrawlableFatLink`	`remove(String url)`
`CrawlableFatLink`	`remove(FatLink fatLink)`
`Boolean`	`hasMore()`
`Integer`	`collectTo(List<UrlAware> sink)`
`List<String>`	`dump()`
`Unit`	`clear()`

Methods inherited from class ai.platon.pulsar.common.collect.collector.AbstractPriorityDataCollector
collectTo, collectTo, collectTo, compareTo
Methods inherited from class ai.platon.pulsar.common.collect.collector.AbstractDataCollector
deepClear
Methods inherited from class ai.platon.pulsar.common.collect.HyperlinkCollector
removeAll, removeAll, toString
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HyperlinkCollector

HyperlinkCollector(PulsarSession session, Queue<NormUrl> seeds, Priority13 priority)

Method Detail

getUrlNormalizer

 final UrlNormalizerPipeline getUrlNormalizer()

setUrlNormalizer

 final Unit setUrlNormalizer(UrlNormalizerPipeline urlNormalizer)

getName
```
 String getName()
```

setName
```
 Unit setName(String name)
```

getSize
```
 Integer getSize()
```

getEstimatedSize
```
 Integer getEstimatedSize()
```

getFatLinks
```
 ConcurrentSkipListMap<String, CrawlableFatLink> getFatLinks()
```
Track the status of this batch, we need a notice when the batch is finished

getSession
```
 final PulsarSession getSession()
```
The pulsar session to use

getSeeds
```
 final Queue<NormUrl> getSeeds()
```
The urls of portal pages from where hyper links are extracted from

getCapacity
```
 Integer getCapacity()
```

getCollectCount
```
 Integer getCollectCount()
```

setCollectCount

 Unit setCollectCount(Integer collectCount)

getCollectTime
```
 Duration getCollectTime()
```

getCollectedCount
```
 Integer getCollectedCount()
```

setCollectedCount

 Unit setCollectedCount(Integer collectedCount)

getCountry
```
 String getCountry()
```

setCountry
```
 Unit setCountry(String country)
```

getCreateTime
```
 Instant getCreateTime()
```

getDeadTime
```
 Instant getDeadTime()
```

setDeadTime
```
 Unit setDeadTime(Instant deadTime)
```

getDistrict
```
 String getDistrict()
```

setDistrict
```
 Unit setDistrict(String district)
```

getEstimatedExternalSize
```
 Integer getEstimatedExternalSize()
```

getExternalSize
```
 Integer getExternalSize()
```

getFirstCollectTime
```
 Instant getFirstCollectTime()
```

setFirstCollectTime

 Unit setFirstCollectTime(Instant firstCollectTime)

getId
```
 Integer getId()
```

getIsDead
```
 Boolean getIsDead()
```

getLabels
```
 Set<String> getLabels()
```

getLang
```
 String getLang()
```

setLang
```
 Unit setLang(String lang)
```

getLastCollectedTime
```
 Instant getLastCollectedTime()
```

setLastCollectedTime

 Unit setLastCollectedTime(Instant lastCollectedTime)

getPriority
```
 Integer getPriority()
```

remove
```
 CrawlableFatLink remove(String url)
```

remove

 CrawlableFatLink remove(FatLink fatLink)

hasMore
```
 Boolean hasMore()
```

collectTo

 Integer collectTo(List<UrlAware> sink)

dump
```
 List<String> dump()
```

clear
```
 Unit clear()
```

Class HyperlinkCollector

Field Summary

Constructor Summary

Method Summary

Methods inherited from class ai.platon.pulsar.common.collect.collector.AbstractPriorityDataCollector

Methods inherited from class ai.platon.pulsar.common.collect.collector.AbstractDataCollector

Methods inherited from class ai.platon.pulsar.common.collect.HyperlinkCollector

Methods inherited from class java.lang.Object

Constructor Detail

HyperlinkCollector

Method Detail

getUrlNormalizer

setUrlNormalizer

getName

setName

getSize

getEstimatedSize

getFatLinks

getSession

getSeeds

getCapacity

getCollectCount

setCollectCount

getCollectTime

getCollectedCount

setCollectedCount

getCountry

setCountry

getCreateTime

getDeadTime

setDeadTime

getDistrict

setDistrict

getEstimatedExternalSize

getExternalSize

getFirstCollectTime

setFirstCollectTime

getId

getIsDead

getLabels

getLang

setLang

getLastCollectedTime

setLastCollectedTime

getPriority

remove

remove

hasMore

collectTo

dump

clear