org.springframework.yarn.batch.partition
Class AbstractPartitionHandler

java.lang.Object
  extended by org.springframework.yarn.batch.partition.AbstractPartitionHandler
All Implemented Interfaces:
PartitionHandler
Direct Known Subclasses:
SplitterPartitionHandler, StaticPartitionHandler

public abstract class AbstractPartitionHandler
extends Object
implements PartitionHandler

Base implementation of Spring Batch PartitionHandler handling partitioning for Yarn containers.

Author:
Janne Valkealahti

Constructor Summary
AbstractPartitionHandler()
          Instantiates a new abstract partition handler.
AbstractPartitionHandler(BatchYarnAppmaster batchAppmaster)
          Instantiates a new abstract partition handler.
 
Method Summary
protected  Map<StepExecution,org.springframework.yarn.am.container.ContainerRequestHint> createRequestData(Set<StepExecution> stepExecutions)
          Subclass may override this method to assign a specific ContainerRequestHint to a StepExecution.
protected abstract  Set<StepExecution> createSplits(StepExecutionSplitter stepSplitter, StepExecution stepExecution)
          Creates the splits.
 String getKeySplitLocations()
          Gets the key split locations.
 String getStepName()
          Gets the remote step name.
 Collection<StepExecution> handle(StepExecutionSplitter stepSplitter, StepExecution stepExecution)
           
 void setBatchAppmaster(BatchYarnAppmaster batchAppmaster)
          Sets the batch appmaster.
 void setKeySplitLocations(String keySplitLocations)
          The name of the key for the split locations in each ExecutionContext.
 void setStepName(String stepName)
          Sets the remote step name.
 void setYarnAppmaster(org.springframework.yarn.am.YarnAppmaster yarnAppmaster)
          Sets the batch appmaster.
protected  void waitCompleteState(StepExecution masterStepExecution)
          Uses CountDownLatch to wait completion status from application master.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

AbstractPartitionHandler

public AbstractPartitionHandler()
Instantiates a new abstract partition handler.


AbstractPartitionHandler

public AbstractPartitionHandler(BatchYarnAppmaster batchAppmaster)
Instantiates a new abstract partition handler.

Parameters:
batchAppmaster - the batch appmaster
Method Detail

handle

public final Collection<StepExecution> handle(StepExecutionSplitter stepSplitter,
                                              StepExecution stepExecution)
                                       throws Exception
Specified by:
handle in interface PartitionHandler
Throws:
Exception

setBatchAppmaster

public void setBatchAppmaster(BatchYarnAppmaster batchAppmaster)
Sets the batch appmaster.

Parameters:
batchAppmaster - the new batch appmaster
See Also:
setYarnAppmaster(YarnAppmaster)

setYarnAppmaster

public void setYarnAppmaster(org.springframework.yarn.am.YarnAppmaster yarnAppmaster)
Sets the batch appmaster. This is a specific method using a YarnAppmaster to be able to allow application context to auto-wire BatchYarnAppmaster if it's type in context is defined as YarnAppmaster.

Parameters:
yarnAppmaster - the new yarn appmaster
See Also:
setBatchAppmaster(BatchYarnAppmaster)

setKeySplitLocations

public void setKeySplitLocations(String keySplitLocations)
The name of the key for the split locations in each ExecutionContext. Defaults to "splitLocations".

Parameters:
keySplitLocations - the value of the key

getKeySplitLocations

public String getKeySplitLocations()
Gets the key split locations.

Returns:
the key split locations

getStepName

public String getStepName()
Gets the remote step name.

Returns:
the remote step name

setStepName

public void setStepName(String stepName)
Sets the remote step name.

Parameters:
stepName - the new remote step name

createSplits

protected abstract Set<StepExecution> createSplits(StepExecutionSplitter stepSplitter,
                                                   StepExecution stepExecution)
                                            throws Exception
Creates the splits. Implementor needs to override this method to create an actual step splits.

Parameters:
stepSplitter - the step splitter
stepExecution - the step execution
Returns:
the step executions
Throws:
Exception - the exception

createRequestData

protected Map<StepExecution,org.springframework.yarn.am.container.ContainerRequestHint> createRequestData(Set<StepExecution> stepExecutions)
                                                                                                   throws Exception
Subclass may override this method to assign a specific ContainerRequestHint to a StepExecution. This would be needed in cases where step should be executed in a specific host or rack considering data locality.

Default implementation returns an empty map.

Parameters:
stepExecutions - Set of step executions
Returns:
Mapping between step executions and container request data
Throws:
Exception - If error occurred

waitCompleteState

protected void waitCompleteState(StepExecution masterStepExecution)
Uses CountDownLatch to wait completion status from application master. Status is considered to be complete if either master itself or parent step execution sends complete status.

Parameters:
masterStepExecution - the parent step execution