T - the type of an entity to writeK - the type of a partition keypublic abstract class AbstractPartitionDataStoreWriter<T,K> extends LifecycleObjectSupport implements PartitionDataStoreWriter<T,K>
PartitionDataStoreWriter.| Constructor and Description |
|---|
AbstractPartitionDataStoreWriter(Configuration configuration,
Path basePath,
CodecInfo codec,
PartitionStrategy<T,K> partitionStrategy)
Instantiates a new abstract data store partition writer.
|
| Modifier and Type | Method and Description |
|---|---|
void |
close() |
protected abstract DataStoreWriter<T> |
createWriter(Configuration configuration,
Path basePath,
CodecInfo codec)
Need to be implemented by a subclass for an actual writer.
|
protected void |
destroyWriter(Path path)
Destroys a writer with a given
Path if exist. |
protected void |
doStart()
Subclasses may implement this method with the start behaviour.
|
protected void |
doStop()
Subclasses may implement this method with the stop behaviour.
|
void |
flush() |
Path |
getBasePath()
Gets the base path.
|
long |
getCloseTimeout()
Gets the close timeout.
|
CodecInfo |
getCodec()
Gets the codec.
|
Configuration |
getConfiguration()
Gets the hadoop configuration.
|
FileNamingStrategyFactory<FileNamingStrategy> |
getFileNamingStrategyFactory()
Gets the file naming strategy factory.
|
long |
getIdleTimeout()
Gets the idle timeout.
|
String |
getInWritingPrefix()
Gets the in writing prefix.
|
String |
getInWritingSuffix()
Gets the in writing suffix.
|
int |
getMaxOpenAttempts()
Gets the max open attempts.
|
RolloverStrategyFactory<RolloverStrategy> |
getRolloverStrategyFactory()
Gets the rollover strategy factory.
|
boolean |
isAppendable()
Checks if append is enabled.
|
boolean |
isClosed()
Checks if this writer is closed.
|
boolean |
isOverwrite()
Checks if overwrite is enabled.
|
boolean |
isSyncable()
Checks if syncable is enabled.
|
protected void |
onInit()
Subclasses may implement this for initialization logic.
|
void |
setAppendable(boolean append)
Set stream as append mode.
|
void |
setCloseTimeout(long closeTimeout)
Sets the close timeout.
|
void |
setFileNamingStrategyFactory(FileNamingStrategyFactory<FileNamingStrategy> fileNamingStrategyFactory)
Sets the file naming strategy factory.
|
void |
setIdleTimeout(long idleTimeout)
Sets the idle timeout.
|
void |
setInWritingPrefix(String prefix)
Sets the in writing prefix.
|
void |
setInWritingSuffix(String suffix)
Sets the in writing suffix.
|
void |
setMaxOpenAttempts(int maxOpenAttempts)
Sets the max open attempts.
|
void |
setOverwrite(boolean overwrite)
Sets the flag indicating if written files may be overwritten.
|
void |
setRolloverStrategyFactory(RolloverStrategyFactory<RolloverStrategy> rolloverStrategyFactory)
Sets the rollover strategy factory.
|
void |
setSyncable(boolean syncable)
Sets the syncable.
|
void |
write(T entity)
Write an entity.
|
void |
write(T entity,
K partitionKey)
Write an entity with an explicit partitioning key.
|
afterPropertiesSet, getBeanFactory, getPhase, getStoreEventPublisher, getTaskExecutor, getTaskScheduler, isAutoStartup, isRunning, setAutoStartup, setBeanFactory, setPhase, setStoreEventPublisher, setTaskExecutor, setTaskScheduler, start, stop, stoppublic AbstractPartitionDataStoreWriter(Configuration configuration, Path basePath, CodecInfo codec, PartitionStrategy<T,K> partitionStrategy)
configuration - the hadoop configurationbasePath - the hdfs pathcodec - the compression codec infopartitionStrategy - the partition strategypublic void write(T entity) throws IOException
DataWriterwrite in interface DataWriter<T>entity - the entity to writeIOException - if an I/O error occurspublic void flush()
throws IOException
flush in interface FlushableIOExceptionpublic void close()
throws IOException
close in interface Closeableclose in interface AutoCloseableIOExceptionpublic void write(T entity, K partitionKey) throws IOException
PartitionDataStoreWriterwrite in interface PartitionDataStoreWriter<T,K>entity - the entity to writepartitionKey - the partition keyIOException - if an I/O error occursprotected void onInit()
throws Exception
LifecycleObjectSupportInitializingBean phase.
Implementor should always call super method not to break initialization chain.onInit in class LifecycleObjectSupportException - if error occurred during initprotected void doStart()
LifecycleObjectSupportLifecycleObjectSupport.lifecycleLock.doStart in class LifecycleObjectSupportprotected void doStop()
LifecycleObjectSupportLifecycleObjectSupport.lifecycleLock.doStop in class LifecycleObjectSupportpublic void setFileNamingStrategyFactory(FileNamingStrategyFactory<FileNamingStrategy> fileNamingStrategyFactory)
fileNamingStrategyFactory - the new file naming strategy factorypublic FileNamingStrategyFactory<FileNamingStrategy> getFileNamingStrategyFactory()
public void setRolloverStrategyFactory(RolloverStrategyFactory<RolloverStrategy> rolloverStrategyFactory)
rolloverStrategyFactory - the new rollover strategy factorypublic RolloverStrategyFactory<RolloverStrategy> getRolloverStrategyFactory()
public void setIdleTimeout(long idleTimeout)
idleTimeout - the new idle timeoutpublic void setCloseTimeout(long closeTimeout)
closeTimeout - the new idle timeoutpublic void setInWritingSuffix(String suffix)
suffix - the new in writing suffixpublic String getInWritingSuffix()
public void setInWritingPrefix(String prefix)
prefix - the new in writing prefixpublic String getInWritingPrefix()
public void setOverwrite(boolean overwrite)
FALSE meaning StoreException
is thrown if file is about to get overwritten.overwrite - the new overwritepublic boolean isOverwrite()
setOverwrite(boolean)public boolean isAppendable()
public void setAppendable(boolean append)
append - the append flagpublic boolean isSyncable()
public void setSyncable(boolean syncable)
syncable - the syncable flagpublic long getIdleTimeout()
public long getCloseTimeout()
public Configuration getConfiguration()
public Path getBasePath()
public CodecInfo getCodec()
public void setMaxOpenAttempts(int maxOpenAttempts)
maxOpenAttempts - the new max open attemptspublic int getMaxOpenAttempts()
public boolean isClosed()
protected abstract DataStoreWriter<T> createWriter(Configuration configuration, Path basePath, CodecInfo codec)
configuration - the configurationbasePath - the base pathcodec - the codec