water.persist

Class PersistManager

java.lang.Object

water.persist.PersistManager

public class PersistManager
extends java.lang.Object

One true persistence manager which hides the implementations from H2O. In particular, HDFS support or S3 support may or may not exist depending on what is on the classpath.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	PersistManager.PersistStatsEntry
static class	PersistManager.Schemes Persistence schemes; used as file prefixes eg "hdfs://some_hdfs_path/some_file"

Field Summary

Fields

Modifier and Type	Field and Description
static int	MAX_BACKENDS

Constructor Summary

Constructors

Constructor and Description
PersistManager(java.net.URI iceRoot)

Method Summary

Methods

Modifier and Type	Method and Description
Key	anyURIToKey(java.net.URI uri) Convert given URI into a specific H2O key representation.
java.util.List<java.lang.String>	calcTypeaheadMatches(java.lang.String filter, int limit) Calculate typeahead matches for src
java.io.OutputStream	create(java.lang.String path, boolean overwrite)
void	delete(int backend, Value v)
boolean	delete(java.lang.String path)
boolean	exists(java.lang.String path)
java.lang.String	getHdfsHomeDirectory()
Persist	getIce() Get the current Persist flavor for user-mode swapping.
Persist	getPersistForURI(java.net.URI uri)
PersistManager.PersistStatsEntry[]	getStats()
void	importFiles(java.lang.String path, java.lang.String pattern, java.util.ArrayList<java.lang.String> files, java.util.ArrayList<java.lang.String> keys, java.util.ArrayList<java.lang.String> fails, java.util.ArrayList<java.lang.String> dels) From a path produce a list of files and keys for parsing.
boolean	isDirectory(java.lang.String path)
boolean	isEmptyDirectoryAllNodes(java.lang.String path) Checks whether a given path is either an empty directory or it doesn't yet exist.
boolean	isHdfsPath(java.lang.String path)
long	length(java.lang.String path)
Persist.PersistEntry[]	list(java.lang.String path)
byte[]	load(int backend, Value v)
java.util.ArrayList<java.lang.String>	matchPattern(java.lang.String prefix, java.util.ArrayList<java.lang.String> fileList, java.lang.String matchStr) Finds all entries in the list that matches the regex
boolean	mkdirs(java.lang.String path)
java.io.InputStream	open(java.lang.String path)
boolean	rename(java.lang.String fromPath, java.lang.String toPath)
void	store(int backend, Value v)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_BACKENDS

public static final int MAX_BACKENDS

See Also:

Constant Field Values

Constructor Detail

PersistManager

public PersistManager(java.net.URI iceRoot)

Method Detail

getStats

public PersistManager.PersistStatsEntry[] getStats()

isHdfsPath

public boolean isHdfsPath(java.lang.String path)

store

public void store(int backend,
         Value v)
           throws java.io.IOException

Throws:

java.io.IOException

delete

public void delete(int backend,
          Value v)

load

public byte[] load(int backend,
          Value v)
            throws java.io.IOException

Throws:

java.io.IOException

getIce

public Persist getIce()

Get the current Persist flavor for user-mode swapping.

anyURIToKey

public final Key anyURIToKey(java.net.URI uri)
                      throws java.io.IOException

Convert given URI into a specific H2O key representation. The representation depends on persistent backend, since it will deduce file location from the key content. The method will look at scheme of URI and based on it, it will ask a backend to provide a conversion to a key (i.e., URI with scheme 'hdfs' will be forwared to HDFS backend).

Parameters:

uri - file location

Returns:

a key encoding URI

Throws:

java.io.IOException - in the case of uri conversion problem

H2OIllegalArgumentException - in case of unsupported scheme

calcTypeaheadMatches

public java.util.List<java.lang.String> calcTypeaheadMatches(java.lang.String filter,
                                                    int limit)

Calculate typeahead matches for src

Parameters:

filter - Source string to match for typeahead

limit - Max number of entries to return

Returns:

List of matches

importFiles

public void importFiles(java.lang.String path,
               java.lang.String pattern,
               java.util.ArrayList<java.lang.String> files,
               java.util.ArrayList<java.lang.String> keys,
               java.util.ArrayList<java.lang.String> fails,
               java.util.ArrayList<java.lang.String> dels)

From a path produce a list of files and keys for parsing. Use as follows: ArrayList files = new ArrayList(); ArrayList keys = new ArrayList(); ArrayList fails = new ArrayList(); ArrayList dels = new ArrayList(); importFiles(importFiles.path, files, keys, fails, dels);

Parameters:

path - (Input) Path to import data from

pattern - (Input) Regex pattern to match files by

files - (Output) List of files found

keys - (Output) List of keys corresponding to files

fails - (Output) List of failed files which mismatch among nodes

dels - (Output) I don't know what this is

getHdfsHomeDirectory

public java.lang.String getHdfsHomeDirectory()

list

public Persist.PersistEntry[] list(java.lang.String path)

exists

public boolean exists(java.lang.String path)

isDirectory

public boolean isDirectory(java.lang.String path)

isEmptyDirectoryAllNodes

public boolean isEmptyDirectoryAllNodes(java.lang.String path)

Checks whether a given path is either an empty directory or it doesn't yet exist. This is trivial if the filesystem where the path leads is distributed. If we are working with a local filesystem we need to make sure that this property is satisfied on all of the nodes.

Parameters:

path - path we want to check

Returns:

true the path is an empty or non-existent directory everywhere, false otherwise

length

public long length(java.lang.String path)

open

public java.io.InputStream open(java.lang.String path)

mkdirs

public boolean mkdirs(java.lang.String path)

rename

public boolean rename(java.lang.String fromPath,
             java.lang.String toPath)

create

public java.io.OutputStream create(java.lang.String path,
                          boolean overwrite)

delete

public boolean delete(java.lang.String path)

getPersistForURI

public Persist getPersistForURI(java.net.URI uri)

matchPattern

public java.util.ArrayList<java.lang.String> matchPattern(java.lang.String prefix,
                                                 java.util.ArrayList<java.lang.String> fileList,
                                                 java.lang.String matchStr)

Finds all entries in the list that matches the regex

Parameters:

prefix - The substring to extract before pattern matching

fileList - The list of strings to check

matchStr - The regular expression to use on the string after prefix

Returns:

list containing the matching entries