BatchFetchComponent

All Implemented Interfaces:: java.lang.AutoCloseable


public final class BatchFetchComponent
extends FetchComponent

Field Summary

Fields
Modifier and Type	Field	Description
`private final GlobalCache`	`globalCache`
`private final WebDb`	`webDb`
`private final GlobalCacheFactory`	`globalCacheFactory`
`private final Boolean`	`isActive`
`private final CoreMetrics`	`coreMetrics`
`private final ProtocolFactory`	`protocolFactory`
`private final ImmutableConfig`	`immutableConfig`

Constructor Summary

Constructors
Constructor	Description
`BatchFetchComponent(WebDb webDb, ImmutableConfig immutableConfig)`
`BatchFetchComponent(WebDb webDb, GlobalCacheFactory globalCacheFactory, CoreMetrics coreMetrics, ProtocolFactory protocolFactory, ImmutableConfig immutableConfig)`

Method Summary

Modifier and Type	Method	Description
`final GlobalCache`	`getGlobalCache()`
`final WebDb`	`getWebDb()`
`final GlobalCacheFactory`	`getGlobalCacheFactory()`
`final Boolean`	`getIsActive()`
`final CoreMetrics`	`getCoreMetrics()`
`final ProtocolFactory`	`getProtocolFactory()`
`final ImmutableConfig`	`getImmutableConfig()`
`final Collection<WebPage>`	`fetchAll(Iterable<String> urls, LoadOptions options)`	Fetch all the urls, config property 'fetch.concurrency' controls the concurrency level.
`final Collection<WebPage>`	`parallelFetchAll(Iterable<String> urls, LoadOptions options)`	Parallel fetch all the urlsEager fetch only some urls to response as soon as possible, the rest urls will be fetched in background later
`final Collection<WebPage>`	`parallelFetchAllGroupedBySchema(Iterable<String> urls, LoadOptions options)`	Group all urls by URL schema, and parallel fetch each groupEager fetch only some urls to response as soon as possible, the rest urls will be fetched in background later

Methods inherited from class ai.platon.pulsar.crawl.component.BatchFetchComponent
close, fetch, fetch, fetchContent, fetchContent, fetchContentDeferred
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - BatchFetchComponent
```
BatchFetchComponent(WebDb webDb, ImmutableConfig immutableConfig)
```
  - BatchFetchComponent
```
BatchFetchComponent(WebDb webDb, GlobalCacheFactory globalCacheFactory, CoreMetrics coreMetrics, ProtocolFactory protocolFactory, ImmutableConfig immutableConfig)
```
- Method Detail
  - getGlobalCache
```
 final GlobalCache getGlobalCache()
```
  - getWebDb
```
 final WebDb getWebDb()
```
  - getGlobalCacheFactory
```
 final GlobalCacheFactory getGlobalCacheFactory()
```
  - getIsActive
```
 final Boolean getIsActive()
```
  - getCoreMetrics
```
 final CoreMetrics getCoreMetrics()
```
  - getProtocolFactory
```
 final ProtocolFactory getProtocolFactory()
```
  - getImmutableConfig
```
 final ImmutableConfig getImmutableConfig()
```
  - fetchAll
```
 final Collection<WebPage> fetchAll(Iterable<String> urls, LoadOptions options)
```
    Fetch all the urls, config property 'fetch.concurrency' controls the concurrency level. If concurrency level is not great than 1, fetch all urls in the caller thread
    Eager fetch only some urls to response as soon as possible, the rest urls will be fetched in background later
    
    Parameters:
    
    urls - The urls to fetch
    
    options - The options
  - parallelFetchAll
```
 final Collection<WebPage> parallelFetchAll(Iterable<String> urls, LoadOptions options)
```
    Parallel fetch all the urls
    Eager fetch only some urls to response as soon as possible, the rest urls will be fetched in background later
    
    Parameters:
    
    urls - The urls to fetch
    
    options - The options
  - parallelFetchAllGroupedBySchema
```
 final Collection<WebPage> parallelFetchAllGroupedBySchema(Iterable<String> urls, LoadOptions options)
```
    Group all urls by URL schema, and parallel fetch each group
    Eager fetch only some urls to response as soon as possible, the rest urls will be fetched in background later
    
    Parameters:
    
    urls - The urls to fetch
    
    options - The options

Class BatchFetchComponent

Field Summary

Constructor Summary

Method Summary

Methods inherited from class ai.platon.pulsar.crawl.component.BatchFetchComponent

Methods inherited from class java.lang.Object

Constructor Detail

BatchFetchComponent

BatchFetchComponent

Method Detail

getGlobalCache

getWebDb

getGlobalCacheFactory

getIsActive

getCoreMetrics

getProtocolFactory

getImmutableConfig

fetchAll

parallelFetchAll

parallelFetchAllGroupedBySchema