AbstractPulsarContext

All Implemented Interfaces:

ai.platon.pulsar.context.PulsarContext , java.lang.AutoCloseable
```
public abstract class AbstractPulsarContext
 implements PulsarContext, AutoCloseable
                    
```
The main entry point for pulsar functionality.
A PulsarContext can be used to inject, fetch, load, parse, store webpages.

Nested Class Summary

Nested Classes
Modifier and Type	Class	Description
`public class`	`AbstractPulsarContext.Companion`

Field Summary

Fields
Modifier and Type	Field	Description
`private final Integer`	`id`
`private final ImmutableConfig`	`unmodifiedConfig`
`private final CrawlUrlNormalizers`	`urlNormalizers`
`private final WebDb`	`webDb`
`private final GlobalCacheFactory`	`globalCacheFactory`
`private final InjectComponent`	`injectComponent`
`private final BatchFetchComponent`	`fetchComponent`
`private final ParseComponent`	`parseComponent`
`private final UpdateComponent`	`updateComponent`
`private final LoadComponent`	`loadComponent`
`private final UrlPool`	`crawlPool`
`private final CrawlLoops`	`crawlLoops`
`private final Long`	`startTime`
`private final Boolean`	`isActive`
`private final ConcurrentSkipListMap<Integer, PulsarSession>`	`sessions`
`private final AbstractApplicationContext`	`applicationContext`
`private final PulsarEnvironment`	`pulsarEnvironment`

Constructor Summary

Constructors
Constructor	Description
`AbstractPulsarContext(AbstractApplicationContext applicationContext, PulsarEnvironment pulsarEnvironment)`

Method Summary

Modifier and Type	Method	Description
`Integer`	`getId()`
`ImmutableConfig`	`getUnmodifiedConfig()`
`CrawlUrlNormalizers`	`getUrlNormalizers()`
`WebDb`	`getWebDb()`
`GlobalCacheFactory`	`getGlobalCacheFactory()`
`InjectComponent`	`getInjectComponent()`
`BatchFetchComponent`	`getFetchComponent()`
`ParseComponent`	`getParseComponent()`
`UpdateComponent`	`getUpdateComponent()`
`LoadComponent`	`getLoadComponent()`
`UrlPool`	`getCrawlPool()`
`CrawlLoops`	`getCrawlLoops()`
`final Long`	`getStartTime()`	The start time
`final Boolean`	`getIsActive()`
`final ConcurrentSkipListMap<Integer, PulsarSession>`	`getSessions()`	All open sessions
`AbstractApplicationContext`	`getApplicationContext()`
`PulsarEnvironment`	`getPulsarEnvironment()`
`final <T extends Any> T`	`getBean(KClass<T> requiredType)`
`final <T extends Any> T`	`getBean()`
`final <T extends Any> T`	`getBeanOrNull(KClass<T> requiredType)`
`final <T extends Any> T`	`getBeanOrNull()`
`abstract AbstractPulsarSession`	`createSession()`
`Unit`	`closeSession(PulsarSession session)`
`Unit`	`registerClosable(AutoCloseable closable)`	Close objects when sessions closes
`final Unit`	`clearCaches()`
`NormUrl`	`normalize(String url, LoadOptions options, Boolean toItemOption)`	Normalize an url, the url can be one of the following: a normal url a configured url a base64 encoded url a base64 encoded configured url An url can be configured by appending arguments to the url, and it also can be used with a LoadOptions, If both tailing arguments and LoadOptions are present, the LoadOptions overrides the tailing arguments, but default values in LoadOptions are ignored.
`List<NormUrl>`	`normalize(Iterable<String> urls, LoadOptions options, Boolean toItemOption)`	Normalize urls, remove invalid ones
`NormUrl`	`normalize(UrlAware url, LoadOptions options, Boolean toItemOption)`	Normalize an url.
`List<NormUrl>`	`normalize(Collection<UrlAware> urls, LoadOptions options, Boolean toItemOption)`	Normalize urls, remove invalid ones
`NormUrl`	`normalizeOrNull(String url, LoadOptions options, Boolean toItemOption)`
`NormUrl`	`normalizeOrNull(UrlAware url, LoadOptions options, Boolean toItemOption)`
`WebPage`	`inject(String url)`	Inject an url
`WebPage`	`inject(NormUrl url)`
`WebPage`	`get(String url)`	Get a webpage from the storage
`WebPage`	`getOrNull(String url)`	Get a webpage from the storage
`Boolean`	`exists(String url)`	Check if a page exists in the storage
`CheckState`	`fetchState(WebPage page, LoadOptions options)`	Check the fetch state of a page
`Iterator<WebPage>`	`scan(String urlPrefix)`	Scan pages in the storage
`Iterator<WebPage>`	`scan(String urlPrefix, Iterable<GWebPage.Field> fields)`	Scan pages in the storage
`Iterator<WebPage>`	`scan(String urlPrefix, Array<String> fields)`	Scan pages in the storage
`WebPage`	`load(String url, LoadOptions options)`	Load an page with specified options, see LoadOptions for all options
`WebPage`	`load(URL url, LoadOptions options)`	Load a url with specified options, see LoadOptions for all options
`WebPage`	`load(NormUrl url)`	Load a url, options can be specified following the url, see LoadOptions for all options
`WebPage`	`loadDeferred(NormUrl url)`
`List<WebPage>`	`loadAll(Iterable<String> urls, LoadOptions options)`	Load a batch of urls with the specified options.
`List<WebPage>`	`loadAll(Iterable<NormUrl> urls)`
`CompletableFuture<WebPage>`	`loadAsync(NormUrl url)`
`List<CompletableFuture<WebPage>>`	`loadAllAsync(Iterable<NormUrl> urls)`
`AbstractPulsarContext`	`submit(UrlAware url)`
`AbstractPulsarContext`	`submitAll(Iterable<UrlAware> urls)`
`FeaturedDocument`	`parse(WebPage page)`	Parse the WebPage using parseComponent
`Unit`	`persist(WebPage page)`	Persist the page into the storage
`Unit`	`delete(String url)`	Delete the page from the storage
`Unit`	`delete(WebPage page)`	Delete the page from the storage
`Unit`	`flush()`	Flush the storage
`Unit`	`await()`	Wait until there is no tasks in the main loop
`Unit`	`registerShutdownHook()`	Register a shutdown hook with the JVM runtime, closing this context on JVM shutdown unless it has already been closed at that time.
`Unit`	`close()`	Close this pulsar contextDelegates to `doClose()` for the actual closing procedure.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - AbstractPulsarContext
```
AbstractPulsarContext(AbstractApplicationContext applicationContext, PulsarEnvironment pulsarEnvironment)
```
- Method Detail
  - getId
```
 Integer getId()
```
  - getUnmodifiedConfig
```
 ImmutableConfig getUnmodifiedConfig()
```
  - getUrlNormalizers
```
 CrawlUrlNormalizers getUrlNormalizers()
```
  - getWebDb
```
 WebDb getWebDb()
```
  - getGlobalCacheFactory
```
 GlobalCacheFactory getGlobalCacheFactory()
```
  - getInjectComponent
```
 InjectComponent getInjectComponent()
```
  - getFetchComponent
```
 BatchFetchComponent getFetchComponent()
```
  - getParseComponent
```
 ParseComponent getParseComponent()
```
  - getUpdateComponent
```
 UpdateComponent getUpdateComponent()
```
  - getLoadComponent
```
 LoadComponent getLoadComponent()
```
  - getCrawlPool
```
 UrlPool getCrawlPool()
```
  - getCrawlLoops
```
 CrawlLoops getCrawlLoops()
```
  - getStartTime
```
 final Long getStartTime()
```
    The start time
  - getIsActive
```
 final Boolean getIsActive()
```
  - getSessions
```
 final ConcurrentSkipListMap<Integer, PulsarSession> getSessions()
```
    All open sessions
  - getApplicationContext
```
 AbstractApplicationContext getApplicationContext()
```
  - getPulsarEnvironment
```
 PulsarEnvironment getPulsarEnvironment()
```
  - getBean
```
 final <T extends Any> T getBean(KClass<T> requiredType)
```
  - getBean
```
 final <T extends Any> T getBean()
```
  - getBeanOrNull
```
 final <T extends Any> T getBeanOrNull(KClass<T> requiredType)
```
  - getBeanOrNull
```
 final <T extends Any> T getBeanOrNull()
```
  - createSession
```
 abstract AbstractPulsarSession createSession()
```
  - closeSession
```
 Unit closeSession(PulsarSession session)
```
  - registerClosable
```
 Unit registerClosable(AutoCloseable closable)
```
    Close objects when sessions closes
  - clearCaches
```
 final Unit clearCaches()
```
  - normalize
```
 NormUrl normalize(String url, LoadOptions options, Boolean toItemOption)
```
    Normalize an url, the url can be one of the following:
    a normal url
    a configured url
    a base64 encoded url
    a base64 encoded configured url
    An url can be configured by appending arguments to the url, and it also can be used with a LoadOptions, If both tailing arguments and LoadOptions are present, the LoadOptions overrides the tailing arguments, but default values in LoadOptions are ignored.
  - normalize
```
 List<NormUrl> normalize(Iterable<String> urls, LoadOptions options, Boolean toItemOption)
```
    Normalize urls, remove invalid ones
    
    Parameters:
    
    urls - The urls to normalize
    
    options - The LoadOptions applied to each url
    
    toItemOption - If the LoadOptions is converted to item load options
  - normalize
```
 NormUrl normalize(UrlAware url, LoadOptions options, Boolean toItemOption)
```
    Normalize an url.
    If both url arguments and LoadOptions are present, the LoadOptions overrides the tailing arguments, but default values in LoadOptions are ignored.
  - normalize
```
 List<NormUrl> normalize(Collection<UrlAware> urls, LoadOptions options, Boolean toItemOption)
```
    Normalize urls, remove invalid ones
    
    Parameters:
    
    urls - The urls to normalize
    
    options - The LoadOptions applied to each url
    
    toItemOption - If the LoadOptions is converted to item load options
  - normalizeOrNull
```
 NormUrl normalizeOrNull(String url, LoadOptions options, Boolean toItemOption)
```
  - normalizeOrNull
```
 NormUrl normalizeOrNull(UrlAware url, LoadOptions options, Boolean toItemOption)
```
  - inject
```
 WebPage inject(String url)
```
    Inject an url
    
    Parameters:
    
    url - The url followed by options
  - inject
```
 WebPage inject(NormUrl url)
```
  - get
```
 WebPage get(String url)
```
    Get a webpage from the storage
  - getOrNull
```
 WebPage getOrNull(String url)
```
    Get a webpage from the storage
  - exists
```
 Boolean exists(String url)
```
    Check if a page exists in the storage
  - fetchState
```
 CheckState fetchState(WebPage page, LoadOptions options)
```
    Check the fetch state of a page
  - scan
```
 Iterator<WebPage> scan(String urlPrefix)
```
    Scan pages in the storage
  - scan
```
 Iterator<WebPage> scan(String urlPrefix, Iterable<GWebPage.Field> fields)
```
    Scan pages in the storage
  - scan
```
 Iterator<WebPage> scan(String urlPrefix, Array<String> fields)
```
    Scan pages in the storage
  - load
```
 WebPage load(String url, LoadOptions options)
```
    Load an page with specified options, see LoadOptions for all options
    
    Parameters:
    
    url - The url followed by options
    
    options - The options
  - load
```
 WebPage load(URL url, LoadOptions options)
```
    Load a url with specified options, see LoadOptions for all options
    
    Parameters:
    
    url - The url followed by options
    
    options - The options
  - load
```
 WebPage load(NormUrl url)
```
    Load a url, options can be specified following the url, see LoadOptions for all options
    
    Parameters:
    
    url - The url followed by options
  - loadDeferred
```
 WebPage loadDeferred(NormUrl url)
```
  - loadAll
```
 List<WebPage> loadAll(Iterable<String> urls, LoadOptions options)
```
    Load a batch of urls with the specified options.
    If the option indicates prefer parallel, urls are fetched in a parallel manner whenever applicable. If the batch is too large, only a random part of the urls is fetched immediately, all the rest urls are put into a pending fetch list and will be fetched in background later.
    If a page exists neither in local storage nor at the given remote location, WebPage.NIL is returned
    
    Parameters:
    
    urls - The urls to load
    
    options - The options
  - loadAll
```
 List<WebPage> loadAll(Iterable<NormUrl> urls)
```
  - loadAsync
```
 CompletableFuture<WebPage> loadAsync(NormUrl url)
```
  - loadAllAsync
```
 List<CompletableFuture<WebPage>> loadAllAsync(Iterable<NormUrl> urls)
```
  - submit
```
 AbstractPulsarContext submit(UrlAware url)
```
  - submitAll
```
 AbstractPulsarContext submitAll(Iterable<UrlAware> urls)
```
  - parse
```
 FeaturedDocument parse(WebPage page)
```
    Parse the WebPage using parseComponent
  - persist
```
 Unit persist(WebPage page)
```
    Persist the page into the storage
  - delete
```
 Unit delete(String url)
```
    Delete the page from the storage
  - delete
```
 Unit delete(WebPage page)
```
    Delete the page from the storage
  - flush
```
 Unit flush()
```
    Flush the storage
  - await
```
 Unit await()
```
    Wait until there is no tasks in the main loop
  - registerShutdownHook
```
 Unit registerShutdownHook()
```
    Register a shutdown hook with the JVM runtime, closing this context on JVM shutdown unless it has already been closed at that time.
    Delegates to doClose() for the actual closing procedure.
  - close
```
 Unit close()
```
    Close this pulsar context
    Delegates to doClose() for the actual closing procedure. Also removes a JVM shutdown hook, if registered, as it's not needed anymore.

Class AbstractPulsarContext

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

AbstractPulsarContext

Method Detail

getId

getUnmodifiedConfig

getUrlNormalizers

getWebDb

getGlobalCacheFactory

getInjectComponent

getFetchComponent

getParseComponent

getUpdateComponent

getLoadComponent

getCrawlPool

getCrawlLoops

getStartTime

getIsActive

getSessions

getApplicationContext

getPulsarEnvironment

getBean

getBean

getBeanOrNull

getBeanOrNull

createSession

closeSession

registerClosable

clearCaches

normalize

normalize

normalize

normalize

normalizeOrNull

normalizeOrNull

inject

inject

get

getOrNull

exists

fetchState

scan

scan

scan

load

load

load

loadDeferred

loadAll

loadAll

loadAsync

loadAllAsync

submit

submitAll

parse

persist

delete

delete

flush

await

registerShutdownHook

close