water.persist

Class PersistManager

java.lang.Object

water.persist.PersistManager

public class PersistManager
extends java.lang.Object

One true persistence manager which hides the implementations from H2O. In particular, HDFS support or S3 support may or may not exist depending on what is on the classpath.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	PersistManager.PersistStatsEntry
static interface	PersistManager.Schemes Persistence schemes; used as file prefixes eg "hdfs://some_hdfs_path/some_file"

Field Summary

Fields

Modifier and Type	Field and Description
static int	MAX_BACKENDS
static int	VALUE_DRIVE

Constructor Summary

Constructors

Constructor and Description
PersistManager(java.net.URI iceRoot)

Method Summary

Modifier and Type	Method and Description
Key	anyURIToKey(java.net.URI uri) Convert given URI into a specific H2O key representation.
java.util.List<java.lang.String>	calcTypeaheadMatches(java.lang.String filter, int limit) Calculate typeahead matches for src
java.io.OutputStream	create(java.lang.String path, boolean overwrite)
void	delete(int backend, Value v)
boolean	delete(java.lang.String path)
boolean	exists(java.lang.String path)
java.lang.String	getHdfsHomeDirectory()
Persist	getIce() Get the current Persist flavor for user-mode swapping.
Persist	getPersistForURI(java.net.URI uri)
PersistManager.PersistStatsEntry[]	getStats()
void	importFiles(java.lang.String[] paths, java.lang.String pattern, java.util.ArrayList<java.lang.String> files, java.util.ArrayList<java.lang.String> keys, java.util.ArrayList<java.lang.String> fails, java.util.ArrayList<java.lang.String> dels)
void	importFiles(java.lang.String path, java.lang.String pattern, java.util.ArrayList<java.lang.String> files, java.util.ArrayList<java.lang.String> keys, java.util.ArrayList<java.lang.String> fails, java.util.ArrayList<java.lang.String> dels) From a path produce a list of files and keys for parsing.
boolean	isEmptyDirectoryAllNodes(java.lang.String path) Checks whether a given path is either an empty directory or it doesn't yet exist.
boolean	isGcsPath(java.lang.String path)
boolean	isHdfsPath(java.lang.String path)
boolean	isHexPath(java.lang.String path)
boolean	isWritableDirectory(java.lang.String path) Check whether given path represents a writable directory.
long	length(java.lang.String path)
Persist.PersistEntry[]	list(java.lang.String path)
byte[]	load(int backend, Key k, long skip, int max)
byte[]	load(int backend, Value v)
boolean	mkdirs(java.lang.String path)
java.io.InputStream	open(java.lang.String path)
java.io.InputStream	openSeekable(Vec vec) Opens a Vec in seekable implementation of an InputStream.
boolean	rename(java.lang.String fromPath, java.lang.String toPath)
void	store(int backend, Value v)
java.lang.String	toHexPath(Key<?> key)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_BACKENDS

public static final int MAX_BACKENDS

See Also:

Constant Field Values

VALUE_DRIVE

public static final int VALUE_DRIVE

See Also:

Constant Field Values

Constructor Detail

PersistManager

public PersistManager(java.net.URI iceRoot)

Method Detail

getStats

public PersistManager.PersistStatsEntry[] getStats()

isHdfsPath

public boolean isHdfsPath(java.lang.String path)

isGcsPath

public boolean isGcsPath(java.lang.String path)

isHexPath

public boolean isHexPath(java.lang.String path)

toHexPath

public java.lang.String toHexPath(Key<?> key)

store

public void store(int backend,
                  Value v)
           throws java.io.IOException

Throws:

java.io.IOException

delete

public void delete(int backend,
                   Value v)

load

public byte[] load(int backend,
                   Value v)
            throws java.io.IOException

Throws:

java.io.IOException

load

public byte[] load(int backend,
                   Key k,
                   long skip,
                   int max)
            throws java.io.IOException

Throws:

java.io.IOException

getIce

public Persist getIce()

Get the current Persist flavor for user-mode swapping.

anyURIToKey

public final Key anyURIToKey(java.net.URI uri)
                      throws java.io.IOException

Convert given URI into a specific H2O key representation. The representation depends on persistent backend, since it will deduce file location from the key content. The method will look at scheme of URI and based on it, it will ask a backend to provide a conversion to a key (i.e., URI with scheme 'hdfs' will be forwared to HDFS backend).

Parameters:

uri - file location

Returns:

a key encoding URI

Throws:

java.io.IOException - in the case of uri conversion problem

H2OIllegalArgumentException - in case of unsupported scheme

calcTypeaheadMatches

public java.util.List<java.lang.String> calcTypeaheadMatches(java.lang.String filter,
                                                             int limit)

Calculate typeahead matches for src

Parameters:

filter - Source string to match for typeahead

limit - Max number of entries to return

Returns:

List of matches

importFiles

public void importFiles(java.lang.String[] paths,
                        java.lang.String pattern,
                        java.util.ArrayList<java.lang.String> files,
                        java.util.ArrayList<java.lang.String> keys,
                        java.util.ArrayList<java.lang.String> fails,
                        java.util.ArrayList<java.lang.String> dels)

importFiles

public void importFiles(java.lang.String path,
                        java.lang.String pattern,
                        java.util.ArrayList<java.lang.String> files,
                        java.util.ArrayList<java.lang.String> keys,
                        java.util.ArrayList<java.lang.String> fails,
                        java.util.ArrayList<java.lang.String> dels)

From a path produce a list of files and keys for parsing. Use as follows: ArrayList files = new ArrayList(); ArrayList keys = new ArrayList(); ArrayList fails = new ArrayList(); ArrayList dels = new ArrayList(); importFiles(importFiles.path, files, keys, fails, dels);

Parameters:

path - (Input) Path to import data from

pattern - (Input) Regex pattern to match files by

files - (Output) List of files found

keys - (Output) List of keys corresponding to files

fails - (Output) List of failed files which mismatch among nodes

dels - (Output) I don't know what this is

getHdfsHomeDirectory

public java.lang.String getHdfsHomeDirectory()

list

public Persist.PersistEntry[] list(java.lang.String path)

exists

public boolean exists(java.lang.String path)

isEmptyDirectoryAllNodes

public boolean isEmptyDirectoryAllNodes(java.lang.String path)

Checks whether a given path is either an empty directory or it doesn't yet exist. This is trivial if the filesystem where the path leads is distributed. If we are working with a local filesystem we need to make sure that this property is satisfied on all the nodes.

Parameters:

path - path we want to check

Returns:

true the path is an empty or non-existent directory everywhere, false otherwise

isWritableDirectory

public boolean isWritableDirectory(java.lang.String path)

Check whether given path represents a writable directory. If such path does not exist it will try to create the directory and if successful it is safe to assume that such directory is writable.

Parameters:

path - Path to check if a writable directory

Returns:

true if given path is a writable directory, false otherwise

length

public long length(java.lang.String path)

open

public java.io.InputStream open(java.lang.String path)

openSeekable

public java.io.InputStream openSeekable(Vec vec)
                                 throws java.io.IOException

Opens a Vec in seekable implementation of an InputStream.

Parameters:

vec - An instance of Vec to open

Returns:

A seekable instanceo of InputStream, never null.

Throws:

java.io.IOException - When the underlying resource does not allow seekable resource creation and all fallback solutions failed. Or on reading error.

mkdirs

public boolean mkdirs(java.lang.String path)

rename

public boolean rename(java.lang.String fromPath,
                      java.lang.String toPath)

create

public java.io.OutputStream create(java.lang.String path,
                                   boolean overwrite)

delete

public boolean delete(java.lang.String path)

getPersistForURI

public Persist getPersistForURI(java.net.URI uri)