-
- All Implemented Interfaces:
-
ai.platon.pulsar.common.config.Parameterized,ai.platon.pulsar.crawl.schedule.FetchSchedule
public final class NewsFetchSchedule extends AdaptiveFetchSchedule
This class implements an adaptive re-fetch algorithm.
NOTE: values of DEC_FACTOR and INC_FACTOR higher than 0.4f may destabilize the algorithm, so that the fetch interval either increases or decreases infinitely, with little relevance to the page changes
-
-
Field Summary
Fields Modifier and Type Field Description private final DurationmaxFetchIntervalprivate final ImmutableConfigconfprivate final MiscMessageWritermessageWriter
-
Constructor Summary
Constructors Constructor Description NewsFetchSchedule(ImmutableConfig conf, MiscMessageWriter messageWriter)
-
Method Summary
Modifier and Type Method Description DurationgetMaxFetchInterval()final ImmutableConfiggetConf()final MiscMessageWritergetMessageWriter()UnitsetFetchSchedule(WebPage page, ModifyInfo m)Sets the fetchIntervalandfetchTimeon a successfully fetched page.-
Methods inherited from class ai.platon.pulsar.crawl.schedule.NewsFetchSchedule
getParams, setPageGoneSchedule -
Methods inherited from class ai.platon.pulsar.crawl.schedule.AdaptiveFetchSchedule
estimatePrevFetchTime, forceRefetch, initializeSchedule, setPageRetrySchedule, shouldFetch -
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
-
Constructor Detail
-
NewsFetchSchedule
NewsFetchSchedule(ImmutableConfig conf, MiscMessageWriter messageWriter)
-
-
Method Detail
-
getMaxFetchInterval
Duration getMaxFetchInterval()
-
getConf
final ImmutableConfig getConf()
-
getMessageWriter
final MiscMessageWriter getMessageWriter()
-
setFetchSchedule
Unit setFetchSchedule(WebPage page, ModifyInfo m)
Sets the
fetchIntervalandfetchTimeon a successfully fetched page. NOTE: this implementation resets the retry counter - extending classes should call super.setFetchSchedule() to preserve this behavior.- Parameters:
page- The Web pagem- The modification info
-
-
-
-