public abstract class AbstractPartitionHandler extends Object implements PartitionHandler
PartitionHandler handling
partitioning for Yarn containers.| Constructor and Description |
|---|
AbstractPartitionHandler()
Instantiates a new abstract partition handler.
|
AbstractPartitionHandler(BatchYarnAppmaster batchAppmaster)
Instantiates a new abstract partition handler.
|
| Modifier and Type | Method and Description |
|---|---|
protected Map<StepExecution,org.springframework.yarn.am.container.ContainerRequestHint> |
createRequestData(Set<StepExecution> stepExecutions)
Subclass may override this method to assign a specific
ContainerRequestHint to
a StepExecution. |
protected abstract Set<StepExecution> |
createSplits(StepExecutionSplitter stepSplitter,
StepExecution stepExecution)
Creates the splits.
|
String |
getKeySplitLocations()
Gets the key split locations.
|
String |
getStepName()
Gets the remote step name.
|
Collection<StepExecution> |
handle(StepExecutionSplitter stepSplitter,
StepExecution stepExecution) |
void |
setBatchAppmaster(BatchYarnAppmaster batchAppmaster)
Sets the batch appmaster.
|
void |
setKeySplitLocations(String keySplitLocations)
The name of the key for the split locations in each
ExecutionContext. |
void |
setStepName(String stepName)
Sets the remote step name.
|
void |
setYarnAppmaster(org.springframework.yarn.am.YarnAppmaster yarnAppmaster)
Sets the batch appmaster.
|
protected void |
waitCompleteState(StepExecution masterStepExecution)
Uses
CountDownLatch to wait completion status from
application master. |
public AbstractPartitionHandler()
public AbstractPartitionHandler(BatchYarnAppmaster batchAppmaster)
batchAppmaster - the batch appmasterpublic final Collection<StepExecution> handle(StepExecutionSplitter stepSplitter, StepExecution stepExecution) throws Exception
handle in interface PartitionHandlerExceptionpublic void setBatchAppmaster(BatchYarnAppmaster batchAppmaster)
batchAppmaster - the new batch appmastersetYarnAppmaster(YarnAppmaster)public void setYarnAppmaster(org.springframework.yarn.am.YarnAppmaster yarnAppmaster)
YarnAppmaster to be able to allow
application context to auto-wire BatchYarnAppmaster
if it's type in context is defined as YarnAppmaster.yarnAppmaster - the new yarn appmastersetBatchAppmaster(BatchYarnAppmaster)public void setKeySplitLocations(String keySplitLocations)
ExecutionContext.
Defaults to "splitLocations".keySplitLocations - the value of the keypublic String getKeySplitLocations()
public String getStepName()
public void setStepName(String stepName)
stepName - the new remote step nameprotected abstract Set<StepExecution> createSplits(StepExecutionSplitter stepSplitter, StepExecution stepExecution) throws Exception
stepSplitter - the step splitterstepExecution - the step executionException - the exceptionprotected Map<StepExecution,org.springframework.yarn.am.container.ContainerRequestHint> createRequestData(Set<StepExecution> stepExecutions) throws Exception
ContainerRequestHint to
a StepExecution. This would be needed in cases where step should be executed
in a specific host or rack considering data locality.
Default implementation returns an empty map.
stepExecutions - Set of step executionsException - If error occurredprotected void waitCompleteState(StepExecution masterStepExecution)
CountDownLatch to wait completion status from
application master. Status is considered to be complete if either
master itself or parent step execution sends complete status.masterStepExecution - the parent step execution