water.persist
Class PersistS3

java.lang.Object
  water.persist.Persist
      water.persist.PersistS3

public final class PersistS3
extends Persist

Persistence backend for S3

Nested Class Summary

static class

PersistS3.H2SO3InputStream

Field Summary

static java.lang.String	S3_CONNECTION_TIMEOUT_PROP S3 connection timeout property name
static java.lang.String	S3_MAX_ERROR_RETRY_PROP S3 maximal error retry number
static java.lang.String	S3_MAX_HTTP_CONNECTIONS_PROP S3 maximal http connections
static java.lang.String	S3_SOCKET_TIMEOUT_PROP S3 socket timeout property name

Fields inherited from class water.persist.Persist

I, UNKNOWN

Constructor Summary

PersistS3()

Method Summary

void	clear()
static java.lang.String[]	decodeKey(Key k) Decodes the given H2O key to the S3 bucket and key name.
void	delete(Value v)
static Key	encodeKey(java.lang.String bucket, java.lang.String key) Creates the key for given S3 bucket and key.
static com.amazonaws.services.s3.AmazonS3	getClient()
java.lang.String	getPath()
Value	lazyArrayChunk(Key key) Lazily manifest data chunks on demand.
byte[]	load(Value v) Value should already be persisted to disk.
void	loadExisting() Load all Key/Value pairs that can be found on the backend.
static Key	loadKey(com.amazonaws.services.s3.model.S3ObjectSummary obj)
static java.io.InputStream	openStream(Key k, Job.ProgressMonitor pmon)
void	store(Value v)

Methods inherited from class water.persist.Persist

getIce, getTotalSpace, getUsableSpace, initialize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

S3_SOCKET_TIMEOUT_PROP

public static final java.lang.String S3_SOCKET_TIMEOUT_PROP

S3 socket timeout property name

See Also:

Constant Field Values

S3_CONNECTION_TIMEOUT_PROP

public static final java.lang.String S3_CONNECTION_TIMEOUT_PROP

S3 connection timeout property name

See Also:

Constant Field Values

S3_MAX_ERROR_RETRY_PROP

public static final java.lang.String S3_MAX_ERROR_RETRY_PROP

S3 maximal error retry number

See Also:

Constant Field Values

S3_MAX_HTTP_CONNECTIONS_PROP

public static final java.lang.String S3_MAX_HTTP_CONNECTIONS_PROP

S3 maximal http connections

See Also:

Constant Field Values

Constructor Detail

PersistS3

public PersistS3()

Method Detail

getClient

public static com.amazonaws.services.s3.AmazonS3 getClient()

openStream

public static java.io.InputStream openStream(Key k,
                                             Job.ProgressMonitor pmon)
                                      throws java.io.IOException

Throws:

java.io.IOException

loadKey

public static Key loadKey(com.amazonaws.services.s3.model.S3ObjectSummary obj)
                   throws java.io.IOException

Throws:

java.io.IOException

load

public byte[] load(Value v)

Description copied from class: Persist

Value should already be persisted to disk. A racing delete can trigger a failure where we get a null return, but no crash (although one could argue that a racing load&delete is a bug no matter what).

Specified by:

load in class Persist

store

public void store(Value v)

Specified by:

store in class Persist

lazyArrayChunk

public Value lazyArrayChunk(Key key)

Description copied from class: Persist

Lazily manifest data chunks on demand.

Specified by:

lazyArrayChunk in class Persist

encodeKey

public static Key encodeKey(java.lang.String bucket,
                            java.lang.String key)

Creates the key for given S3 bucket and key. Returns the H2O key, or null if the key cannot be created.

Parameters:

bucket - Bucket name

key - Key name (S3)

Returns:

H2O key pointing to the given bucket and key.

decodeKey

public static java.lang.String[] decodeKey(Key k)

Decodes the given H2O key to the S3 bucket and key name. Returns the array of two strings, first one is the bucket name and second one is the key name.

Parameters:

k - Key to be decoded.

Returns:

Pair (array) of bucket name and key name.

getPath

public java.lang.String getPath()

Specified by:

getPath in class Persist

clear

public void clear()

Specified by:

clear in class Persist

loadExisting

public void loadExisting()

Description copied from class: Persist

Load all Key/Value pairs that can be found on the backend.

Specified by:

loadExisting in class Persist

delete

public void delete(Value v)

Specified by:

delete in class Persist