Data In H2O¶
A H2OFrame represents a 2D array of data where each column is uniformly typed.
The data may be local or it may be in an H2O cluster. The data are loaded from a CSV file or from a native python data structure, and is either a python-process-local file, a cluster-local file, or a list of H2OVec objects.
Loading Data From A CSV File¶
H2O’s parser supports data of various formats from multiple sources. The following formats are supported:
- SVMLight
- CSV (data may delimited by any of the 128 ASCII characters)
- XLS
- The following data sources are supported:
- NFS / Local File / List of Files
- HDFS
- URL
- A Directory (with many data files inside at the same level – no support for recursive import of data)
- S3/S3N
- Native Language Data Structure (c.f. the subsequent section)
>>> trainFrame = h2o.import_frame(path="hdfs://192.168.1.10/user/data/data_test.csv") #or >>> trainFrame = h2o.import_frame(path="~/data/data_test.csv")
Loading Data From A Python Object¶
To transfer the data that are stored in python data structures to H2O, use the H2OFrame constructor and the python_obj argument. If the python_obj argument is not None, then additional arguments are ignored.
The following types are permissible for python_obj:
- tuple ()
- list []
- dict {}
- collections.OrderedDict
The type of python_obj is inspected by performing an isinstance call. A ValueError will be raised if the type of python_obj is not one of the above types. For example, sets, byte arrays, and un-contained types are not permissible.
The subsequent sections discuss each data type in detail in terms of the “source” representation (the python object) and the “target” representation (the H2O object). Concretely, the topics of discussion will be on the following: Headers, Data Types, Number of Rows, Number of Columns, and Missing Values.
Aside: Why is Pandas’ DataFrame not a permissible type?
There are two reasons that Pandas’ DataFrame objects are not included. First, it is best to minimize the number of dependencies, and secondly it is difficult to justify including the Pandas module as a dependency if its only function is related to this small detail of transferring data from python to H2O.
Second, Pandas objects are simple wrappers of numpy arrays that include some meta data, so the transfer of data from a Pandas DataFrame to an H2O Frame could readily be achieved.
In the following documentation, H2OFrame and Frame will be used synonymously. Technically, an H2OFrame is the object-pointer that resides in the python VM and points to a Frame object inside of the H2O JVM. Similarly, H2OFrame, Frame, and H2O Frame all refer to the same kind of object. In general, though, the context is from the python VM, unless otherwise specified.
Loading A Python Tuple¶
Essentially, the tuple is an immutable list. This immutability does not map to the H2OFrame. So pythonistas beware!
The restrictions on what goes inside the tuple are fairly relaxed, but if they are not recognized, a ValueError displays.
A tuple is formatted as follows:
(i1, i2, i3, ..., iN)
Restrictions are mainly on the types of the individual iJ (1 <= J <= N).
If iJ is {} for some J, then a ValueError displays.
If iJ is a () (tuple) or [] (list), then iJ must be a () or [] for all J; otherwise a ValueError displays.
If iJ is a () or [], and if it is a nested () or nested [], then a ValueError displays. In other words, only a single level of nesting is valid and all internal arrays must be flat – H2O does not flatten them for you.
If iJ is not a () or [], then it must be of type string or a non-complex numeric type (float or int). In other words, if iJ is not a tuple, list, string, float, or int, for some J, then a ValueError displays.
- Some examples of acceptable inputs are:
- Example A: (1,2,3)
- Example B: ((1,2,3), (4,5,6), (“cat”, “dog”))
- Example C: ((1,2,3), [4,5,6], [“blue”, “yellow”], (321.239, “green”,”hi”))
- Example D: (3284.123891, “dog”, 89)
Note that it is perfectly fine to mix () and [] within a tuple.
Headers, Columns, Rows, Data Types, and Missing Values:
The format of the H2OFrame is as follows:
column1 column2 column3 ... columnN a11, a12, a13, ..., a1N ., ., ., ..., . ., ., ., ..., . ., ., ., ..., . aM1, aM2, aM3, ..., aMN
It looks exactly like an MxN matrix with an additional header “row”. This header cannot be specified when loading data from a () (or from a [] but it is possible to specify a header with a python dictionary (see below for details).
Headers:
Since no header row can be specified for this case, H2O automatically generates a column header in the following format:
C1, C2, C3, ..., CN
Notably, these columns have a 1-based indexing (i.e. the 0th column is “C1”).
Rows, Columns, and Missing Data:
The shape of the H2OFrame is determined by two factors:
- the number of arrays nested in the ()
- the number of items in each array
If there are no nested arrays (as in Example A and Example D above), the resulting H2OFrame will have the following shape (rows x cols):
1 x len(tuple)
(i.e. a Frame with a single row).
If there are nested arrays (as in Example B and Example C above), then (given the rules stated above) the resulting H2OFrame will have ROWS equal to the number of arrays nested within and COLUMNS equal to the maximum sub-array:
max( [len(l) for l in tuple] ) x len(tuple)
Note that this addresses the issue with ragged sub-arrays by assuming that shorter sub-arrays will pad themselves with NA (missing values) at the end so that they become the correct length.
Because the Frame is uniformly typed, combining data types within a column may produce unexpected results. Please read up on the H2O parser for details on how a column type is determined for mixed-type columns.
Loading A Python List¶
The same principles that apply to tuples also apply to lists. Lists are mutable objects, so there is no semantic difference regarding mutability between an H2OFrame and a list (as there is for a tuple).
Additionally, a list [] is ordered the same way as a tuple (), with the data appearing within the brackets.
Loading A Python Dictionary Or collections.OrderedDict¶
Each entry in the {} is expected to represent a single column. Keys in the {} must be character strings following the pattern: ^[a-zA-Z_][a-zA-Z0-9_.]*$ without restriction on length. A valid column name may begin with any letter (capital or not) or an “_”, followed by any number of letters, digits, “_”s, or ”.”s.
Values in the {} may be a flat [], a flat (), or a single int, float, or string value. Nested [] and () will raise a ValueError. This is the only additional restriction on [] and () that applies in this context.
Note that the built-in dict does not provide any guarantees on ordering. This has implications on the order of columns in the eventual H2OFrame, since they may be written out of order from which they were initially put into the dict.
collections.OrderedDict preserves the order of the key-value pairs in which they were entered.
H2OFrame¶
- class h2o.frame.H2OFrame(python_obj=None, local_fname=None, remote_fname=None, vecs=None, text_key=None)[source]¶
-
- col_names()[source]¶
Retrieve the column names (one name per H2OVec) for this H2OFrame.
Returns: A character list[] of column names.
- ddply(cols, fun)[source]¶
Parameters: - cols – Column names used to control grouping
- fun – Function to execute on each group. Right now limited to textual Rapids expression
Returns: New frame with 1 row per-group, of results from ‘fun’
- describe()[source]¶
Generate an in-depth description of this H2OFrame.
The description is a tabular print of the type, min, max, sigma, number of zeros, and number of missing elements for each H2OVec in this H2OFrame.
Returns: None (print to stdout)
- dim()[source]¶
Get the number of rows and columns in the H2OFrame.
Returns: The number of rows and columns in the H2OFrame as a list [rows, cols].
- drop(i)[source]¶
Column selection via integer, string(name) returns a Vec Column selection via slice returns a subset Frame
Parameters: i – Column to select Returns: Returns an H2OVec or H2OFrame.
- filterNACols(frac=0.2)[source]¶
Filter columns with prportion of NAs >= frac. :param frac: Fraction of NAs in the column. :return: A list of column indices.
- group_by(cols, a, order_by=None)[source]¶
GroupBy :param cols: The columns to group on. :param a: A dictionary of aggregates having the following shape: {“colname”:[aggregate, column, naMethod]} e.g.: {“bikes”:[“count”, 0, “all”]} The naMethod is one of “all”, “ignore”, or “rm”, which specifies how to handle NAs that appear in columns that are being aggregated.
“all” - include NAs “rm” - exclude NAs “ignore” - ignore NAs in aggregates, but count them (e.g. in denominators for mean, var, sd, etc.) :param order_by: A list of column names or indices on which to order the results. :return: The group by frame.
- head(rows=10, cols=200, **kwargs)[source]¶
Analgous to R’s head call on a data.frame. Display a digestible chunk of the H2OFrame starting from the beginning.
Parameters: - rows – Number of rows to display.
- cols – Number of columns to display.
- kwargs – Extra arguments passed from other methods.
Returns: None
- impute(column, method, combine_method, by, inplace)[source]¶
Impute a column in this H2OFrame.
Parameters: - column – The column to impute
- method – How to compute the imputation value.
- combine_method – For even samples and method=”median”, how to combine quantiles.
- by – Columns to group-by for computing imputation value per groups of columns.
- inplace – Impute inplace?
Returns: the imputed frame.
- insert_missing_values(fraction=0.1, seed=None)[source]¶
Inserting Missing Values to an H2OFrame This is primarily used for testing. Randomly replaces a user-specified fraction of entries in a H2O dataset with missing values. WARNING: This will modify the original dataset. Unless this is intended, this function should only be called on a subset of the original.
Parameters: - fraction – A number between 0 and 1 indicating the fraction of entries to replace with missing.
- seed – A random number used to select which entries to replace with missing values. Default of seed = -1 will
automatically generate a seed in H2O. :return: H2OFrame with missing values inserted
- keys()[source]¶
Retrieve the keys for each of the H2OVec objects comrpising this H2OFrame.
Returns: the array of keys.
- levels(col=0)[source]¶
Get the factor levels for this frame and the specified column index.
Parameters: col – A column index in this H2OFrame. Returns: a list of strings that are the factor levels for the column.
- merge(other, allLeft=False, allRite=False)[source]¶
Merge two datasets based on common column names
Parameters: - other – Other dataset to merge. Must have at least one column in common with self, and all columns in common are used as the merge key. If you want to use only a subset of the columns in common, rename the other columns so the columns are unique in the merged result.
- allLeft – If true, include all rows from the left/self frame
- allRite – If true, include all rows from the right/other frame
Returns: Original self frame enhanced with merged columns and rows
- names()[source]¶
Retrieve the column names (one name per H2OVec) for this H2OFrame.
Returns: A character list[] of column names.
- ncol()[source]¶
Get the number of columns in this H2OFrame.
Returns: The number of columns in this H2OFrame.
- nrow()[source]¶
Get the number of rows in this H2OFrame.
Returns: The number of rows in this dataset.
- quantile(prob=None, combine_method='interpolate')[source]¶
Compute quantiles over a given H2OFrame.
Parameters: - prob – A list of probabilties, default is [0.01,0.1,0.25,0.333,0.5,0.667,0.75,0.9,0.99]. You may provide any sequence of any length.
- combine_method – For even samples, how to combine quantiles. Should be one of [“interpolate”, “average”, “low”, “hi”]
Returns: an H2OFrame containing the quantiles and probabilities.
- setNames(names)[source]¶
Change the column names to names.
Parameters: names – A list of strings equal to the number of columns in the H2OFrame. Returns: None. Rename the column names in this H2OFrame.
H2OVec¶
- class h2o.frame.H2OVec(name, expr)[source]¶
A single column of data that is uniformly typed and possibly lazily computed.
- append(data)[source]¶
Append a value during CSV read, convert to float.
Parameters: data – An element being appended to the end of this H2OVec Returns: void
- cbind(data)[source]¶
Parameters: data – H2OFrame or H2OVec Returns: new H2OFrame with data cbinded to the end
- get_expr()[source]¶
Helper method to obtain the expr object in self. Can also get it directly @ ._expr.
Returns: the _expr member of this H2OVec
- isfactor()[source]¶
Returns: A lazy Expr representing the truth of whether or not this vec is a factor.
- static mktime(year=1970, month=0, day=0, hour=0, minute=0, second=0, msec=0)[source]¶
All units are zero-based (including months and days). Missing year is 1970.
Returns: Returns msec since the Epoch.
- quantile(prob=None, combine_method='interpolate')[source]¶
Returns: A lazy Expr representing the quantiles of this H2OVec.
- row_select(vec)[source]¶
Boolean column select lookup
Parameters: vec – An H2OVec. Returns: A new H2OVec.
- runif(seed=None)[source]¶
Parameters: seed – A random seed. If None, then one will be generated. Returns: A new H2OVec filled with doubles sampled uniformly from [0,1).
- setName(name)[source]¶
Set the column name for this column.
Parameters: name – The new name for this column. Returns: None
- show(noprint=False)[source]¶
Pretty print this H2OVec, or return values up to an iterator on an enclosing Frame
Parameters: noprint – A boolean stating whether to print or to return data. Returns: If noprint is False, then self._expr is returned.