|
Spring for Apache Hadoop | ||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD | ||||||||
java.lang.Objectorg.springframework.yarn.fs.DefaultResourceLocalizer
public class DefaultResourceLocalizer
Default implementation of ResourceLocalizer which
is only capable of re-using files already in HDFS and preparing
correct parameters for created LocalResource entries.
| Constructor Summary | |
|---|---|
DefaultResourceLocalizer(org.apache.hadoop.conf.Configuration configuration,
java.util.Collection<LocalResourcesFactoryBean.TransferEntry> transferEntries,
java.util.Collection<LocalResourcesFactoryBean.CopyEntry> copyEntries)
Instantiates a new default resource localizer. |
|
DefaultResourceLocalizer(org.apache.hadoop.conf.Configuration configuration,
java.util.Collection<LocalResourcesFactoryBean.TransferEntry> transferEntries,
java.util.Collection<LocalResourcesFactoryBean.CopyEntry> copyEntries,
org.apache.hadoop.fs.Path stagingDirectory)
Instantiates a new default resource localizer. |
|
| Method Summary | |
|---|---|
boolean |
clean()
Cleans all leftovers what has been created during the distribute process. |
void |
distribute()
If underlying implementation needs to do operations on hdfs filesystem or any other preparation work, calling of this method should make implementation ready to return resources from ResourceLocalizer.getResources()
command. |
protected void |
doFileCopy(org.apache.hadoop.fs.FileSystem fs)
Do file copy. |
protected java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.records.LocalResource> |
doFileTransfer(org.apache.hadoop.fs.FileSystem fs)
Gets a map of localized resources. |
java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.records.LocalResource> |
getResources()
Gets a map of LocalResource instances. |
void |
setStagingDirectory(org.apache.hadoop.fs.Path stagingDirectory)
Sets the staging directory. |
void |
setStagingId(java.lang.String stagingId)
Sets the staging id. |
| Methods inherited from class java.lang.Object |
|---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
| Constructor Detail |
|---|
public DefaultResourceLocalizer(org.apache.hadoop.conf.Configuration configuration,
java.util.Collection<LocalResourcesFactoryBean.TransferEntry> transferEntries,
java.util.Collection<LocalResourcesFactoryBean.CopyEntry> copyEntries)
configuration - the configurationtransferEntries - the transfer entriescopyEntries - the copy entries
public DefaultResourceLocalizer(org.apache.hadoop.conf.Configuration configuration,
java.util.Collection<LocalResourcesFactoryBean.TransferEntry> transferEntries,
java.util.Collection<LocalResourcesFactoryBean.CopyEntry> copyEntries,
org.apache.hadoop.fs.Path stagingDirectory)
configuration - the configurationtransferEntries - the transfer entriescopyEntries - the copy entriesstagingDirectory - the staging directory| Method Detail |
|---|
public java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.records.LocalResource> getResources()
ResourceLocalizerLocalResource instances. Underlying
instances of LocalResources needs to be fully
initialised including resource size and timestamp.
getResources in interface ResourceLocalizerLocalResource instancespublic void setStagingDirectory(org.apache.hadoop.fs.Path stagingDirectory)
ResourceLocalizer
setStagingDirectory in interface ResourceLocalizerstagingDirectory - the new staging directorypublic void setStagingId(java.lang.String stagingId)
ResourceLocalizerPath
set in ResourceLocalizer.setStagingDirectory(Path) to post fix unique
runtime staging path. If not set simultaneous instances of
same application may override files.
setStagingId in interface ResourceLocalizerstagingId - the new staging idpublic void distribute()
ResourceLocalizerResourceLocalizer.getResources()
command.
distribute in interface ResourceLocalizerpublic boolean clean()
ResourceLocalizer
clean in interface ResourceLocalizer
protected void doFileCopy(org.apache.hadoop.fs.FileSystem fs)
throws java.io.IOException
fs - the fs
java.io.IOException - Signals that an I/O exception has occurred.
protected java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.records.LocalResource> doFileTransfer(org.apache.hadoop.fs.FileSystem fs)
throws java.io.IOException,
java.net.URISyntaxException
fs - the file system
java.io.IOException - if problem occurred getting file status
java.net.URISyntaxException - if file path is wrong
|
Spring for Apache Hadoop | ||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD | ||||||||