Method Detail

• createSession

   abstract PulsarSession createSession()

• closeSession

   abstract Unit closeSession(PulsarSession session)

• registerClosable

   abstract Unit registerClosable(AutoCloseable closable)

  Close objects when sessions close

• normalize

   abstract NormUrl normalize(String url, LoadOptions options, Boolean toItemOption)

• normalize

   abstract List<NormUrl> normalize(Iterable<String> urls, LoadOptions options, Boolean toItemOption)

• normalize

   abstract NormUrl normalize(UrlAware url, LoadOptions options, Boolean toItemOption)

• normalize

   abstract List<NormUrl> normalize(Collection<UrlAware> urls, LoadOptions options, Boolean toItemOption)

• normalizeOrNull

   abstract NormUrl normalizeOrNull(String url, LoadOptions options, Boolean toItemOption)

• normalizeOrNull

   abstract NormUrl normalizeOrNull(UrlAware url, LoadOptions options, Boolean toItemOption)

• inject

   abstract WebPage inject(String url)

  Inject an url

  Parameters:

  url - The url followed by config options

• inject

   abstract WebPage inject(NormUrl url)

• get

   abstract WebPage get(String url)

• getOrNull

   abstract WebPage getOrNull(String url)

• exists

   abstract Boolean exists(String url)

• fetchState

   abstract CheckState fetchState(WebPage page, LoadOptions options)

• scan

   abstract Iterator<WebPage> scan(String urlPrefix)

• scan

   abstract Iterator<WebPage> scan(String urlPrefix, Iterable<GWebPage.Field> fields)

• scan

   abstract Iterator<WebPage> scan(String urlPrefix, Array<String> fields)

• load

   abstract WebPage load(String url, LoadOptions options)

  Load a url with specified options, see LoadOptions for all options

  Parameters:

  url - The url followed by options

  options - The options

• load

   abstract WebPage load(URL url, LoadOptions options)

  Load a url with specified options, see LoadOptions for all options

  Parameters:

  url - The url followed by options

  options - The options

• load

   abstract WebPage load(NormUrl url)

  Load a url, options can be specified following the url, see LoadOptions for all options

  Parameters:

  url - The url followed by options

• loadDeferred

   abstract WebPage loadDeferred(NormUrl url)

• loadAll

   abstract Collection<WebPage> loadAll(Iterable<String> urls, LoadOptions options)

  Load a batch of urls with the specified options.

  If the option indicates prefer parallel, urls are fetched in a parallel manner whenever applicable. If the batch is too large, only a random part of the urls is fetched immediately, all the rest urls are put into a pending fetch list and will be fetched in background later.

  If a page exists neither in local storage nor at the given remote location, WebPage.NIL is returned

  Parameters:

  urls - The urls to load

  options - The options

• loadAll

   abstract Collection<WebPage> loadAll(Iterable<NormUrl> urls)

• loadAsync

   abstract CompletableFuture<WebPage> loadAsync(NormUrl url)

• loadAllAsync

   abstract List<CompletableFuture<WebPage>> loadAllAsync(Iterable<NormUrl> urls)

• submit

   abstract PulsarContext submit(UrlAware url)

• submitAll

   abstract PulsarContext submitAll(Iterable<UrlAware> urls)

• parse

   abstract FeaturedDocument parse(WebPage page)

  Parse the WebPage using Jsoup

• persist

   abstract Unit persist(WebPage page)

• delete

   abstract Unit delete(String url)

• delete

   abstract Unit delete(WebPage page)

• flush

   abstract Unit flush()

• await

   abstract Unit await()

  Wait until all tasks are done.

• registerShutdownHook

   abstract Unit registerShutdownHook()

• getId

   abstract Integer getId()

• getPulsarEnvironment

   abstract PulsarEnvironment getPulsarEnvironment()

• getApplicationContext

   abstract ApplicationContext getApplicationContext()

• getUnmodifiedConfig

   abstract ImmutableConfig getUnmodifiedConfig()

• getCrawlPool

   abstract UrlPool getCrawlPool()

• getCrawlLoops

   abstract CrawlLoops getCrawlLoops()