org.springframework.yarn.batch.partition
Class AbstractBatchPartitionHandler

java.lang.Object
  extended by org.springframework.yarn.batch.partition.AbstractBatchPartitionHandler
All Implemented Interfaces:
PartitionHandler
Direct Known Subclasses:
HdfsSplitBatchPartitionHandler, StaticBatchPartitionHandler

public abstract class AbstractBatchPartitionHandler
extends Object
implements PartitionHandler

Base implementation of Spring Batch PartitionHandler handling partitioning for Yarn containers.

Author:
Janne Valkealahti

Constructor Summary
AbstractBatchPartitionHandler()
          Instantiates a new batch partition handler.
AbstractBatchPartitionHandler(AbstractBatchAppmaster batchAppmaster)
          Instantiates a new batch partition handler.
 
Method Summary
protected  Map<StepExecution,org.springframework.yarn.am.container.ContainerRequestHint> createResourceRequestData(Set<StepExecution> stepExecutions)
          Subclass may override this method to assign a specific ContainerRequestHint to a StepExecution.
protected abstract  Set<StepExecution> createStepExecutionSplits(StepExecutionSplitter stepSplitter, StepExecution stepExecution)
           
 org.springframework.yarn.am.container.ContainerResolver getContainerResolver()
          Gets the container resolver.
 String getStepName()
          Gets the step name.
 Collection<StepExecution> handle(StepExecutionSplitter stepSplitter, StepExecution stepExecution)
           
 void setBatchAppmaster(AbstractBatchAppmaster batchAppmaster)
          Sets the batch appmaster.
 void setContainerResolver(org.springframework.yarn.am.container.ContainerResolver containerResolver)
          Sets the container resolver.
 void setStepName(String stepName)
          Sets the step name.
protected  void waitCompleteState(StepExecution masterStepExecution)
          Uses CountDownLatch to wait completion status from application master.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

AbstractBatchPartitionHandler

public AbstractBatchPartitionHandler()
Instantiates a new batch partition handler.


AbstractBatchPartitionHandler

public AbstractBatchPartitionHandler(AbstractBatchAppmaster batchAppmaster)
Instantiates a new batch partition handler.

Parameters:
batchAppmaster - the batch appmaster
Method Detail

setBatchAppmaster

public void setBatchAppmaster(AbstractBatchAppmaster batchAppmaster)
Sets the batch appmaster.

Parameters:
batchAppmaster - the new batch appmaster

createStepExecutionSplits

protected abstract Set<StepExecution> createStepExecutionSplits(StepExecutionSplitter stepSplitter,
                                                                StepExecution stepExecution)
                                                         throws Exception
Throws:
Exception

createResourceRequestData

protected Map<StepExecution,org.springframework.yarn.am.container.ContainerRequestHint> createResourceRequestData(Set<StepExecution> stepExecutions)
                                                                                                           throws Exception
Subclass may override this method to assign a specific ContainerRequestHint to a StepExecution. This would be needed in cases where step should be executed in a specific host or rack considering data locality. Default implementation returns an empty map.

Parameters:
stepExecutions - Set of step executions
Returns:
Mapping between step executions and container request data
Throws:
Exception - If error occurred

handle

public final Collection<StepExecution> handle(StepExecutionSplitter stepSplitter,
                                              StepExecution stepExecution)
                                       throws Exception
Specified by:
handle in interface PartitionHandler
Throws:
Exception

getStepName

public String getStepName()
Gets the step name.

Returns:
the step name

setStepName

public void setStepName(String stepName)
Sets the step name.

Parameters:
stepName - the new step name

getContainerResolver

public org.springframework.yarn.am.container.ContainerResolver getContainerResolver()
Gets the container resolver.

Returns:
the container resolver

setContainerResolver

public void setContainerResolver(org.springframework.yarn.am.container.ContainerResolver containerResolver)
Sets the container resolver.

Parameters:
containerResolver - the new container resolver

waitCompleteState

protected void waitCompleteState(StepExecution masterStepExecution)
Uses CountDownLatch to wait completion status from application master. Status is considered to be complete if either master itself or parent step execution sends complete status.

Parameters:
masterStepExecution - the parent step execution