-
- All Implemented Interfaces:
-
ai.platon.pulsar.common.StartStopRunnable,ai.platon.pulsar.crawl.CrawlLoop
public abstract class AbstractCrawlLoop implements CrawlLoop
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description public classAbstractCrawlLoop.Companion
-
Field Summary
Fields Modifier and Type Field Description private final Integeridprivate LoadOptionsdefaultOptionsprivate final UrlFeederurlFeederprivate final List<PriorityDataCollector<UrlAware>>collectorsprivate final AbstractCrawlercrawlerprivate final Stringabstractprivate final Stringreportprivate final Stringnameprivate final ImmutableConfigconfig
-
Constructor Summary
Constructors Constructor Description AbstractCrawlLoop(String name, ImmutableConfig config)
-
Method Summary
Modifier and Type Method Description IntegergetId()LoadOptionsgetDefaultOptions()Data collector lower capacity UnitsetDefaultOptions(LoadOptions defaultOptions)Data collector lower capacity abstract UrlFeedergetUrlFeeder()The fetch iterable from which all fetch tasks are taken List<PriorityDataCollector<UrlAware>>getCollectors()abstract AbstractCrawlergetCrawler()StringgetAbstract()StringgetReport()StringgetName()ImmutableConfiggetConfig()-
-
Constructor Detail
-
AbstractCrawlLoop
AbstractCrawlLoop(String name, ImmutableConfig config)
-
-
Method Detail
-
getDefaultOptions
LoadOptions getDefaultOptions()
Data collector lower capacity
-
setDefaultOptions
Unit setDefaultOptions(LoadOptions defaultOptions)
Data collector lower capacity
-
getUrlFeeder
abstract UrlFeeder getUrlFeeder()
The fetch iterable from which all fetch tasks are taken
-
getCollectors
List<PriorityDataCollector<UrlAware>> getCollectors()
-
getCrawler
abstract AbstractCrawler getCrawler()
-
getAbstract
String getAbstract()
-
getConfig
ImmutableConfig getConfig()
-
-
-
-