-
- All Implemented Interfaces:
-
java.lang.AutoCloseable
public final class BatchFetchComponent extends FetchComponent
-
-
Field Summary
Fields Modifier and Type Field Description private final GlobalCacheglobalCacheprivate final WebDbwebDbprivate final GlobalCacheFactoryglobalCacheFactoryprivate final BooleanisActiveprivate final CoreMetricscoreMetricsprivate final ProtocolFactoryprotocolFactoryprivate final ImmutableConfigimmutableConfig
-
Constructor Summary
Constructors Constructor Description BatchFetchComponent(WebDb webDb, ImmutableConfig immutableConfig)BatchFetchComponent(WebDb webDb, GlobalCacheFactory globalCacheFactory, CoreMetrics coreMetrics, ProtocolFactory protocolFactory, ImmutableConfig immutableConfig)
-
Method Summary
Modifier and Type Method Description final GlobalCachegetGlobalCache()final WebDbgetWebDb()final GlobalCacheFactorygetGlobalCacheFactory()final BooleangetIsActive()final CoreMetricsgetCoreMetrics()final ProtocolFactorygetProtocolFactory()final ImmutableConfiggetImmutableConfig()final Collection<WebPage>fetchAll(Iterable<String> urls, LoadOptions options)Fetch all the urls, config property 'fetch.concurrency' controls the concurrency level. final Collection<WebPage>parallelFetchAll(Iterable<String> urls, LoadOptions options)Parallel fetch all the urlsEager fetch only some urls to response as soon as possible, the rest urls will be fetched in background later final Collection<WebPage>parallelFetchAllGroupedBySchema(Iterable<String> urls, LoadOptions options)Group all urls by URL schema, and parallel fetch each groupEager fetch only some urls to response as soon as possible, the rest urls will be fetched in background later -
-
Constructor Detail
-
BatchFetchComponent
BatchFetchComponent(WebDb webDb, ImmutableConfig immutableConfig)
-
BatchFetchComponent
BatchFetchComponent(WebDb webDb, GlobalCacheFactory globalCacheFactory, CoreMetrics coreMetrics, ProtocolFactory protocolFactory, ImmutableConfig immutableConfig)
-
-
Method Detail
-
getGlobalCache
final GlobalCache getGlobalCache()
-
getWebDb
final WebDb getWebDb()
-
getGlobalCacheFactory
final GlobalCacheFactory getGlobalCacheFactory()
-
getIsActive
final Boolean getIsActive()
-
getCoreMetrics
final CoreMetrics getCoreMetrics()
-
getProtocolFactory
final ProtocolFactory getProtocolFactory()
-
getImmutableConfig
final ImmutableConfig getImmutableConfig()
-
fetchAll
final Collection<WebPage> fetchAll(Iterable<String> urls, LoadOptions options)
Fetch all the urls, config property 'fetch.concurrency' controls the concurrency level. If concurrency level is not great than 1, fetch all urls in the caller thread
Eager fetch only some urls to response as soon as possible, the rest urls will be fetched in background later
- Parameters:
urls- The urls to fetchoptions- The options
-
parallelFetchAll
final Collection<WebPage> parallelFetchAll(Iterable<String> urls, LoadOptions options)
Parallel fetch all the urls
Eager fetch only some urls to response as soon as possible, the rest urls will be fetched in background later
- Parameters:
urls- The urls to fetchoptions- The options
-
parallelFetchAllGroupedBySchema
final Collection<WebPage> parallelFetchAllGroupedBySchema(Iterable<String> urls, LoadOptions options)
Group all urls by URL schema, and parallel fetch each group
Eager fetch only some urls to response as soon as possible, the rest urls will be fetched in background later
- Parameters:
urls- The urls to fetchoptions- The options
-
-
-
-