GenerateComponent

All Implemented Interfaces:

ai.platon.pulsar.common.config.Parameterized , ai.platon.pulsar.crawl.common.JobInitialized
```
public final class GenerateComponent
 implements Parameterized, JobInitialized
                    
```
Parser checker, useful for testing parser. It also accurately reports possible fetching and parsing failures and presents protocol status signals to aid debugging. The tool enables us to retrieve the following data from any

Nested Class Summary

Nested Classes
Modifier and Type	Class	Description
`public class`	`GenerateComponent.Companion`

Field Summary

Fields
Modifier and Type	Field	Description
`private final Logger`	`LOG`
`private final CrawlFilters`	`crawlFilters`
`private final WebDb`	`webDb`
`private final CrawlUrlFilters`	`urlFilters`
`private final CrawlUrlNormalizers`	`urlNormalizers`
`private final FetchSchedule`	`fetchSchedule`
`private final MiscMessageWriter`	`messageWriter`
`private final ImmutableConfig`	`conf`

Constructor Summary

Constructors
Constructor	Description
`GenerateComponent(CrawlFilters crawlFilters, WebDb webDb, CrawlUrlFilters urlFilters, CrawlUrlNormalizers urlNormalizers, FetchSchedule fetchSchedule, MiscMessageWriter messageWriter, ImmutableConfig conf)`

Method Summary

Modifier and Type	Method	Description
`final Logger`	`getLOG()`
`final CrawlFilters`	`getCrawlFilters()`
`final WebDb`	`getWebDb()`
`final CrawlUrlFilters`	`getUrlFilters()`
`final CrawlUrlNormalizers`	`getUrlNormalizers()`
`final FetchSchedule`	`getFetchSchedule()`
`final MiscMessageWriter`	`getMessageWriter()`
`final ImmutableConfig`	`getConf()`
`Unit`	`setup(ImmutableConfig jobConf)`
`Params`	`getParams()`
`final Pair<Boolean, String>`	`shouldFetch(String url, String reversedUrl, WebPage page)`	TODO : We may move some filters to hbase query filters directly TODO : Move to CrawlFilter

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GenerateComponent

GenerateComponent(CrawlFilters crawlFilters, WebDb webDb, CrawlUrlFilters urlFilters, CrawlUrlNormalizers urlNormalizers, FetchSchedule fetchSchedule, MiscMessageWriter messageWriter, ImmutableConfig conf)

Method Detail

getLOG
```
 final Logger getLOG()
```

getCrawlFilters
```
 final CrawlFilters getCrawlFilters()
```

getWebDb
```
 final WebDb getWebDb()
```

getUrlFilters
```
 final CrawlUrlFilters getUrlFilters()
```

getUrlNormalizers

 final CrawlUrlNormalizers getUrlNormalizers()

getFetchSchedule
```
 final FetchSchedule getFetchSchedule()
```

getMessageWriter

 final MiscMessageWriter getMessageWriter()

getConf
```
 final ImmutableConfig getConf()
```

setup
```
 Unit setup(ImmutableConfig jobConf)
```

getParams
```
 Params getParams()
```

shouldFetch
```
 final Pair<Boolean, String> shouldFetch(String url, String reversedUrl, WebPage page)
```
TODO : We may move some filters to hbase query filters directly TODO : Move to CrawlFilter

Class GenerateComponent

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

GenerateComponent

Method Detail

getLOG

getCrawlFilters

getWebDb

getUrlFilters

getUrlNormalizers

getFetchSchedule

getMessageWriter

getConf

setup

getParams

shouldFetch