public abstract class AbstractPartitionHandler extends java.lang.Object implements PartitionHandler
PartitionHandler handling
partitioning for Yarn containers.| Constructor and Description |
|---|
AbstractPartitionHandler()
Instantiates a new abstract partition handler.
|
AbstractPartitionHandler(BatchYarnAppmaster batchAppmaster)
Instantiates a new abstract partition handler.
|
| Modifier and Type | Method and Description |
|---|---|
protected java.util.Map<StepExecution,org.springframework.yarn.am.container.ContainerRequestHint> |
createRequestData(java.util.Set<StepExecution> stepExecutions)
Subclass may override this method to assign a specific
ContainerRequestHint to
a StepExecution. |
protected abstract java.util.Set<StepExecution> |
createSplits(StepExecutionSplitter stepSplitter,
StepExecution stepExecution)
Creates the splits.
|
java.lang.String |
getKeySplitLocations()
Gets the key split locations.
|
java.lang.String |
getStepName()
Gets the remote step name.
|
java.util.Collection<StepExecution> |
handle(StepExecutionSplitter stepSplitter,
StepExecution stepExecution) |
void |
setBatchAppmaster(BatchYarnAppmaster batchAppmaster)
Sets the batch appmaster.
|
void |
setKeySplitLocations(java.lang.String keySplitLocations)
The name of the key for the split locations in each
ExecutionContext. |
void |
setStepName(java.lang.String stepName)
Sets the remote step name.
|
void |
setYarnAppmaster(org.springframework.yarn.am.YarnAppmaster yarnAppmaster)
Sets the batch appmaster.
|
protected void |
waitCompleteState(StepExecution masterStepExecution)
Uses
CountDownLatch to wait completion status from
application master. |
public AbstractPartitionHandler()
public AbstractPartitionHandler(BatchYarnAppmaster batchAppmaster)
batchAppmaster - the batch appmasterpublic final java.util.Collection<StepExecution> handle(StepExecutionSplitter stepSplitter, StepExecution stepExecution) throws java.lang.Exception
handle in interface PartitionHandlerjava.lang.Exceptionpublic void setBatchAppmaster(BatchYarnAppmaster batchAppmaster)
batchAppmaster - the new batch appmastersetYarnAppmaster(YarnAppmaster)public void setYarnAppmaster(org.springframework.yarn.am.YarnAppmaster yarnAppmaster)
YarnAppmaster to be able to allow
application context to auto-wire BatchYarnAppmaster
if it's type in context is defined as YarnAppmaster.yarnAppmaster - the new yarn appmastersetBatchAppmaster(BatchYarnAppmaster)public void setKeySplitLocations(java.lang.String keySplitLocations)
ExecutionContext.
Defaults to "splitLocations".keySplitLocations - the value of the keypublic java.lang.String getKeySplitLocations()
public java.lang.String getStepName()
public void setStepName(java.lang.String stepName)
stepName - the new remote step nameprotected abstract java.util.Set<StepExecution> createSplits(StepExecutionSplitter stepSplitter, StepExecution stepExecution) throws java.lang.Exception
stepSplitter - the step splitterstepExecution - the step executionjava.lang.Exception - the exceptionprotected java.util.Map<StepExecution,org.springframework.yarn.am.container.ContainerRequestHint> createRequestData(java.util.Set<StepExecution> stepExecutions) throws java.lang.Exception
ContainerRequestHint to
a StepExecution. This would be needed in cases where step should be executed
in a specific host or rack considering data locality.
Default implementation returns an empty map.
stepExecutions - Set of step executionsjava.lang.Exception - If error occurredprotected void waitCompleteState(StepExecution masterStepExecution)
CountDownLatch to wait completion status from
application master. Status is considered to be complete if either
master itself or parent step execution sends complete status.masterStepExecution - the parent step execution