Serialized Form

Package org.apache.pulsar.io.hdfs3

Class org.apache.pulsar.io.hdfs3.AbstractHdfsConfig extends java.lang.Object implements Serializable

serialVersionUID:

1L

Serialized Fields

compression

Compression compression

The compression codec used to compress/de-compress the files on HDFS.

directory

java.lang.String directory

The HDFS directory from which files should be read from or written to.

encoding

java.lang.String encoding

The character encoding for the files, e.g. UTF-8, ASCII, etc.

hdfsConfigResources

java.lang.String hdfsConfigResources

A file or comma separated list of files which contains the Hadoop file system configuration, e.g. 'core-site.xml', 'hdfs-site.xml'.

kerberosUserPrincipal

java.lang.String kerberosUserPrincipal

The Kerberos user principal account to use for authentication.

keytab

java.lang.String keytab

The full pathname to the Kerberos keytab file to use for authentication.

Package org.apache.pulsar.io.hdfs3.sink

Class org.apache.pulsar.io.hdfs3.sink.HdfsSinkConfig extends AbstractHdfsConfig implements Serializable

serialVersionUID:

1L

Serialized Fields

fileExtension

java.lang.String fileExtension

The extension to add to the files written to HDFS, e.g. '.txt', '.seq', etc.

filenamePrefix

java.lang.String filenamePrefix

The prefix of the files to create inside the HDFS directory, i.e. a value of "topicA" will result in files named topicA-, topicA-, etc being produced

maxPendingRecords

int maxPendingRecords

The maximum number of records that we hold in memory before acking. Default is Integer.MAX_VALUE. Setting this value to one, results in every record being sent to disk before the record is acked, while setting it to a higher values allows us to buffer records before flushing them all to disk.

separator

char separator

The character to use to separate records in a text file. If no value is provided then the content from all of the records will be concatenated together in one continuous byte array.

syncInterval

long syncInterval

The interval (in milliseconds) between calls to flush data to HDFS disk.