net.snowflake.client.jdbc.cloud.storage

Class SnowflakeGCSClient

java.lang.Object

net.snowflake.client.jdbc.cloud.storage.SnowflakeGCSClient

All Implemented Interfaces:

SnowflakeStorageClient

public class SnowflakeGCSClient
extends Object
implements SnowflakeStorageClient

Encapsulates the GCS Storage client and all GCS operations and logic

Author:

ppaulus

Method Summary

Modifier and Type	Method and Description
void	addDigestMetadata(StorageObjectMetadata meta, String digest) Adds digest metadata to the StorageObjectMetadata object
void	addEncryptionMetadata(StorageObjectMetadata meta, MatDesc matDesc, byte[] ivData, byte[] encKeK, long contentLength) Adds encryption metadata to the StorageObjectMetadata object
void	addStreamingIngestMetadata(StorageObjectMetadata meta, String clientName, String clientKey) Adds streaming ingest metadata to the StorageObjectMetadata object, used for streaming ingest per client billing calculation
static SnowflakeGCSClient	createSnowflakeGCSClient(StageInfo stage, net.snowflake.common.core.RemoteStoreFileEncryptionMaterial encMat, SFSession session)
void	download(SFSession session, String command, String localLocation, String destFileName, int parallelism, String remoteStorageLocation, String stageFilePath, String stageRegion, String presignedUrl) Download a file from remote storage.
InputStream	downloadToStream(SFSession session, String command, int parallelism, String remoteStorageLocation, String stageFilePath, String stageRegion, String presignedUrl) Download a file from remote storage
String	getDigestMetadata(StorageObjectMetadata meta) Gets digest metadata to the StorageObjectMetadata object
int	getEncryptionKeySize()
String	getMatdescKey() Returns the material descriptor key
int	getMaxRetries()
StorageObjectMetadata	getObjectMetadata(String remoteStorageLocation, String prefix) Returns the metadata properties for a remote storage object
int	getRetryBackoffMaxExponent() Returns the max exponent for multiplying backoff with the power of 2, the value of 4 will give us 16secs as the max number of time to sleep before retry
int	getRetryBackoffMin()
String	getStreamingIngestClientKey(StorageObjectMetadata meta) Gets streaming ingest client key to the StorageObjectMetadata object
String	getStreamingIngestClientName(StorageObjectMetadata meta) Gets streaming ingest client name to the StorageObjectMetadata object
void	handleStorageException(Exception ex, int retryCount, String operation, SFSession session, String command) Handles exceptions thrown by the remote storage provider
boolean	isEncrypting()
StorageObjectSummaryCollection	listObjects(String remoteStorageLocation, String prefix) listObjects gets all the objects in a path
void	renew(Map<?,?> stageCredentials) Re-creates the encapsulated storage client with a fresh access token
boolean	requirePresignedUrl()
void	shutdown() shuts down the client
void	upload(SFSession session, String command, int parallelism, boolean uploadFromStream, String remoteStorageLocation, File srcFile, String destFileName, InputStream inputStream, FileBackedOutputStream fileBackedOutputStream, StorageObjectMetadata meta, String stageRegion, String presignedUrl) Upload a file/stream to remote storage
void	uploadWithPresignedUrlWithoutConnection(int networkTimeoutInMilli, HttpClientSettingsKey ocspModeAndProxyKey, int parallelism, boolean uploadFromStream, String remoteStorageLocation, File srcFile, String destFileName, InputStream inputStream, FileBackedOutputStream fileBackedOutputStream, StorageObjectMetadata meta, String stageRegion, String presignedUrl) Upload a file (-stream) to remote storage with Pre-signed URL without JDBC session.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

createSnowflakeGCSClient

public static SnowflakeGCSClient createSnowflakeGCSClient(StageInfo stage,
                                                          net.snowflake.common.core.RemoteStoreFileEncryptionMaterial encMat,
                                                          SFSession session)
                                                   throws SnowflakeSQLException

Throws:

SnowflakeSQLException

getMaxRetries

public int getMaxRetries()

Specified by:

getMaxRetries in interface SnowflakeStorageClient

Returns:

Returns the Max number of retry attempts

getRetryBackoffMaxExponent

public int getRetryBackoffMaxExponent()

Description copied from interface: SnowflakeStorageClient

Returns the max exponent for multiplying backoff with the power of 2, the value of 4 will give us 16secs as the max number of time to sleep before retry

Specified by:

getRetryBackoffMaxExponent in interface SnowflakeStorageClient

Returns:

Returns the exponent

getRetryBackoffMin

public int getRetryBackoffMin()

Specified by:

getRetryBackoffMin in interface SnowflakeStorageClient

Returns:

Returns the min number of milliseconds to sleep before retry

isEncrypting

public boolean isEncrypting()

Specified by:

isEncrypting in interface SnowflakeStorageClient

Returns:

Returns true if encryption is enabled

getEncryptionKeySize

public int getEncryptionKeySize()

Specified by:

getEncryptionKeySize in interface SnowflakeStorageClient

Returns:

Returns the size of the encryption key

requirePresignedUrl

public boolean requirePresignedUrl()

Specified by:

requirePresignedUrl in interface SnowflakeStorageClient

Returns:

Whether this client requires the use of presigned URLs for upload and download instead of credentials that work for all files uploaded/ downloaded to a stage path. True for GCS.

renew

public void renew(Map<?,?> stageCredentials)
           throws SnowflakeSQLException

Description copied from interface: SnowflakeStorageClient

Re-creates the encapsulated storage client with a fresh access token

Specified by:

renew in interface SnowflakeStorageClient

Parameters:

stageCredentials - a Map (as returned by GS) which contains the new credential properties

Throws:

SnowflakeSQLException - failure to renew the storage client

shutdown

public void shutdown()

Description copied from interface: SnowflakeStorageClient

shuts down the client

Specified by:

shutdown in interface SnowflakeStorageClient

listObjects

public StorageObjectSummaryCollection listObjects(String remoteStorageLocation,
                                                  String prefix)
                                           throws StorageProviderException

listObjects gets all the objects in a path

Specified by:

listObjects in interface SnowflakeStorageClient

Parameters:

remoteStorageLocation - bucket name

prefix - Path

Returns:

Throws:

StorageProviderException

getObjectMetadata

public StorageObjectMetadata getObjectMetadata(String remoteStorageLocation,
                                               String prefix)
                                        throws StorageProviderException

Description copied from interface: SnowflakeStorageClient

Returns the metadata properties for a remote storage object

Specified by:

getObjectMetadata in interface SnowflakeStorageClient

Parameters:

remoteStorageLocation - location, i.e. bucket for S3

prefix - the prefix/path of the object to retrieve

Returns:

storage metadata object

Throws:

StorageProviderException - cloud storage provider error

download

public void download(SFSession session,
                     String command,
                     String localLocation,
                     String destFileName,
                     int parallelism,
                     String remoteStorageLocation,
                     String stageFilePath,
                     String stageRegion,
                     String presignedUrl)
              throws SnowflakeSQLException

Download a file from remote storage.

Specified by:

download in interface SnowflakeStorageClient

Parameters:

session - session object

command - command to download file

localLocation - local file path

destFileName - destination file name

parallelism - [ not used by the GCP implementation ]

remoteStorageLocation - remote storage location, i.e. bucket for S3

stageFilePath - stage file path

stageRegion - region name where the stage persists

presignedUrl - Credential to use for download

Throws:

SnowflakeSQLException - download failure

downloadToStream

public InputStream downloadToStream(SFSession session,
                                    String command,
                                    int parallelism,
                                    String remoteStorageLocation,
                                    String stageFilePath,
                                    String stageRegion,
                                    String presignedUrl)
                             throws SnowflakeSQLException

Download a file from remote storage

Specified by:

downloadToStream in interface SnowflakeStorageClient

Parameters:

session - session object

command - command to download file

parallelism - number of threads for parallel downloading

remoteStorageLocation - remote storage location, i.e. bucket for s3

stageFilePath - stage file path

stageRegion - region name where the stage persists

presignedUrl - Signed credential for download

Returns:

input file stream

Throws:

SnowflakeSQLException - when download failure

uploadWithPresignedUrlWithoutConnection

public void uploadWithPresignedUrlWithoutConnection(int networkTimeoutInMilli,
                                                    HttpClientSettingsKey ocspModeAndProxyKey,
                                                    int parallelism,
                                                    boolean uploadFromStream,
                                                    String remoteStorageLocation,
                                                    File srcFile,
                                                    String destFileName,
                                                    InputStream inputStream,
                                                    FileBackedOutputStream fileBackedOutputStream,
                                                    StorageObjectMetadata meta,
                                                    String stageRegion,
                                                    String presignedUrl)
                                             throws SnowflakeSQLException

Upload a file (-stream) to remote storage with Pre-signed URL without JDBC session.

Specified by:

uploadWithPresignedUrlWithoutConnection in interface SnowflakeStorageClient

Parameters:

networkTimeoutInMilli - Network timeout for the upload

ocspModeAndProxyKey - OCSP mode and proxy settings for the upload.

parallelism - number of threads do parallel uploading

uploadFromStream - true if upload source is stream

remoteStorageLocation - s3 bucket name

srcFile - source file if not uploading from a stream

destFileName - file name on remote storage after upload

inputStream - stream used for uploading if fileBackedOutputStream is null

fileBackedOutputStream - stream used for uploading if not null

meta - object meta data

stageRegion - region name where the stage persists

presignedUrl - presigned URL for upload. Used by GCP.

Throws:

SnowflakeSQLException - if upload failed

upload

public void upload(SFSession session,
                   String command,
                   int parallelism,
                   boolean uploadFromStream,
                   String remoteStorageLocation,
                   File srcFile,
                   String destFileName,
                   InputStream inputStream,
                   FileBackedOutputStream fileBackedOutputStream,
                   StorageObjectMetadata meta,
                   String stageRegion,
                   String presignedUrl)
            throws SnowflakeSQLException

Upload a file/stream to remote storage

Specified by:

upload in interface SnowflakeStorageClient

Parameters:

session - session object

command - upload command

parallelism - [ not used by the GCP implementation ]

uploadFromStream - true if upload source is stream

remoteStorageLocation - storage container name

srcFile - source file if not uploading from a stream

destFileName - file name on remote storage after upload

inputStream - stream used for uploading if fileBackedOutputStream is null

fileBackedOutputStream - stream used for uploading if not null

meta - object meta data

stageRegion - region name where the stage persists

presignedUrl - Credential used for upload of a file

Throws:

SnowflakeSQLException - if upload failed even after retry

handleStorageException

public void handleStorageException(Exception ex,
                                   int retryCount,
                                   String operation,
                                   SFSession session,
                                   String command)
                            throws SnowflakeSQLException

Description copied from interface: SnowflakeStorageClient

Handles exceptions thrown by the remote storage provider

Specified by:

handleStorageException in interface SnowflakeStorageClient

Parameters:

ex - the exception to handle

retryCount - current number of retries, incremented by the caller before each call

operation - string that indicates the function/operation that was taking place, when the exception was raised, for example "upload"

session - the current SFSession object used by the client

command - the command attempted at the time of the exception

Throws:

SnowflakeSQLException - exceptions that were not handled, or retried past what the retry policy allows, are propagated

getMatdescKey

public String getMatdescKey()

Returns the material descriptor key

Specified by:

getMatdescKey in interface SnowflakeStorageClient

Returns:

the material descriptor key

addEncryptionMetadata

public void addEncryptionMetadata(StorageObjectMetadata meta,
                                  MatDesc matDesc,
                                  byte[] ivData,
                                  byte[] encKeK,
                                  long contentLength)

Adds encryption metadata to the StorageObjectMetadata object

Specified by:

addEncryptionMetadata in interface SnowflakeStorageClient

Parameters:

meta - the storage metadata object to add the encyption info to

matDesc - the material decriptor

ivData - the initialization vector

encKeK - the key encryption key

contentLength - the length of the encrypted content

addDigestMetadata

public void addDigestMetadata(StorageObjectMetadata meta,
                              String digest)

Adds digest metadata to the StorageObjectMetadata object

Specified by:

addDigestMetadata in interface SnowflakeStorageClient

Parameters:

meta - the storage metadata object to add the digest to

digest - the digest metadata to add

getDigestMetadata

public String getDigestMetadata(StorageObjectMetadata meta)

Gets digest metadata to the StorageObjectMetadata object

Specified by:

getDigestMetadata in interface SnowflakeStorageClient

Parameters:

meta - the metadata object to extract the digest metadata from

Returns:

the digest metadata value

addStreamingIngestMetadata

public void addStreamingIngestMetadata(StorageObjectMetadata meta,
                                       String clientName,
                                       String clientKey)

Adds streaming ingest metadata to the StorageObjectMetadata object, used for streaming ingest per client billing calculation

Specified by:

addStreamingIngestMetadata in interface SnowflakeStorageClient

Parameters:

meta - the storage metadata object to add the digest to

clientName - streaming ingest client name

clientKey - streaming ingest client key, provided by Snowflake

getStreamingIngestClientName

public String getStreamingIngestClientName(StorageObjectMetadata meta)

Gets streaming ingest client name to the StorageObjectMetadata object

Specified by:

getStreamingIngestClientName in interface SnowflakeStorageClient

getStreamingIngestClientKey

public String getStreamingIngestClientKey(StorageObjectMetadata meta)

Gets streaming ingest client key to the StorageObjectMetadata object

Specified by:

getStreamingIngestClientKey in interface SnowflakeStorageClient