org.springframework.yarn.batch.partition
Class HdfsSplitBatchPartitionHandler

java.lang.Object
  extended by org.springframework.yarn.batch.partition.AbstractBatchPartitionHandler
      extended by org.springframework.yarn.batch.partition.HdfsSplitBatchPartitionHandler
All Implemented Interfaces:
PartitionHandler

public class HdfsSplitBatchPartitionHandler
extends AbstractBatchPartitionHandler

Implementation of Spring Batch PartitionHandler which does partitioning based on number of input files from HDFS.

Author:
Janne Valkealahti

Constructor Summary
HdfsSplitBatchPartitionHandler(AbstractBatchAppmaster batchAppmaster)
          Instantiates a new hdfs split batch partition handler.
HdfsSplitBatchPartitionHandler(AbstractBatchAppmaster batchAppmaster, Configuration configuration)
          Instantiates a new hdfs split batch partition handler.
 
Method Summary
protected  Map<StepExecution,org.springframework.yarn.am.container.ContainerRequestHint> createResourceRequestData(Set<StepExecution> stepExecutions)
          Subclass may override this method to assign a specific ContainerRequestHint to a StepExecution.
protected  Set<StepExecution> createStepExecutionSplits(StepExecutionSplitter stepSplitter, StepExecution stepExecution)
           
 Configuration getConfiguration()
          Gets the Yarn configuration.
 void setConfiguration(Configuration configuration)
          Sets the Yarn configuration.
 
Methods inherited from class org.springframework.yarn.batch.partition.AbstractBatchPartitionHandler
getContainerResolver, getStepName, handle, setContainerResolver, setStepName, waitCompleteState
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

HdfsSplitBatchPartitionHandler

public HdfsSplitBatchPartitionHandler(AbstractBatchAppmaster batchAppmaster)
Instantiates a new hdfs split batch partition handler.

Parameters:
batchAppmaster - the batch appmaster

HdfsSplitBatchPartitionHandler

public HdfsSplitBatchPartitionHandler(AbstractBatchAppmaster batchAppmaster,
                                      Configuration configuration)
Instantiates a new hdfs split batch partition handler.

Parameters:
batchAppmaster - the batch appmaster
Method Detail

getConfiguration

public Configuration getConfiguration()
Gets the Yarn configuration.

Returns:
the Yarn configuration

setConfiguration

public void setConfiguration(Configuration configuration)
Sets the Yarn configuration.

Parameters:
configuration - the new Yarn configuration

createStepExecutionSplits

protected Set<StepExecution> createStepExecutionSplits(StepExecutionSplitter stepSplitter,
                                                       StepExecution stepExecution)
                                                throws Exception
Specified by:
createStepExecutionSplits in class AbstractBatchPartitionHandler
Throws:
Exception

createResourceRequestData

protected Map<StepExecution,org.springframework.yarn.am.container.ContainerRequestHint> createResourceRequestData(Set<StepExecution> stepExecutions)
                                                                                                           throws Exception
Description copied from class: AbstractBatchPartitionHandler
Subclass may override this method to assign a specific ContainerRequestHint to a StepExecution. This would be needed in cases where step should be executed in a specific host or rack considering data locality. Default implementation returns an empty map.

Overrides:
createResourceRequestData in class AbstractBatchPartitionHandler
Parameters:
stepExecutions - Set of step executions
Returns:
Mapping between step executions and container request data
Throws:
Exception - If error occurred