|
Spring for Apache Hadoop | ||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD | ||||||||
java.lang.Objectorg.springframework.yarn.batch.partition.AbstractBatchPartitionHandler
org.springframework.yarn.batch.partition.HdfsSplitBatchPartitionHandler
public class HdfsSplitBatchPartitionHandler
Implementation of Spring Batch PartitionHandler which does
partitioning based on number of input files from HDFS.
| Constructor Summary | |
|---|---|
HdfsSplitBatchPartitionHandler(AbstractBatchAppmaster batchAppmaster)
Instantiates a new hdfs split batch partition handler. |
|
HdfsSplitBatchPartitionHandler(AbstractBatchAppmaster batchAppmaster,
org.apache.hadoop.conf.Configuration configuration)
Instantiates a new hdfs split batch partition handler. |
|
| Method Summary | |
|---|---|
protected java.util.Map<org.springframework.batch.core.StepExecution,ContainerRequestHint> |
createResourceRequestData(java.util.Set<org.springframework.batch.core.StepExecution> stepExecutions)
Subclass may override this method to assign a specific ContainerRequestHint to
a StepExecution. |
protected java.util.Set<org.springframework.batch.core.StepExecution> |
createStepExecutionSplits(org.springframework.batch.core.partition.StepExecutionSplitter stepSplitter,
org.springframework.batch.core.StepExecution stepExecution)
|
org.apache.hadoop.conf.Configuration |
getConfiguration()
Gets the Yarn configuration. |
void |
setConfiguration(org.apache.hadoop.conf.Configuration configuration)
Sets the Yarn configuration. |
| Methods inherited from class org.springframework.yarn.batch.partition.AbstractBatchPartitionHandler |
|---|
getContainerResolver, getStepName, handle, setContainerResolver, setStepName, waitCompleteState |
| Methods inherited from class java.lang.Object |
|---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
| Constructor Detail |
|---|
public HdfsSplitBatchPartitionHandler(AbstractBatchAppmaster batchAppmaster)
batchAppmaster - the batch appmaster
public HdfsSplitBatchPartitionHandler(AbstractBatchAppmaster batchAppmaster,
org.apache.hadoop.conf.Configuration configuration)
batchAppmaster - the batch appmaster| Method Detail |
|---|
public org.apache.hadoop.conf.Configuration getConfiguration()
public void setConfiguration(org.apache.hadoop.conf.Configuration configuration)
configuration - the new Yarn configuration
protected java.util.Set<org.springframework.batch.core.StepExecution> createStepExecutionSplits(org.springframework.batch.core.partition.StepExecutionSplitter stepSplitter,
org.springframework.batch.core.StepExecution stepExecution)
throws java.lang.Exception
createStepExecutionSplits in class AbstractBatchPartitionHandlerjava.lang.Exception
protected java.util.Map<org.springframework.batch.core.StepExecution,ContainerRequestHint> createResourceRequestData(java.util.Set<org.springframework.batch.core.StepExecution> stepExecutions)
throws java.lang.Exception
AbstractBatchPartitionHandlerContainerRequestHint to
a StepExecution. This would be needed in cases where step should be executed
in a specific host or rack considering data locality. Default implementation
returns an empty map.
createResourceRequestData in class AbstractBatchPartitionHandlerstepExecutions - Set of step executions
java.lang.Exception - If error occurred
|
Spring for Apache Hadoop | ||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD | ||||||||