water.util

Class FrameUtils

java.lang.Object

water.util.FrameUtils

public class FrameUtils
extends java.lang.Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FrameUtils.CalculateWeightMeanSTD This class will calculate the weighted mean and standard deviatioin of a target column of a data frame with the weights specified in another column.
static class	FrameUtils.CategoricalBinaryEncoder Helper to convert a categorical variable into a "binary" encoding format.
static class	FrameUtils.CategoricalEigenEncoder Helper to convert a categorical variable into the first eigenvector of the dummy-expanded matrix.
static class	FrameUtils.CategoricalEnumLimitedEncoder Helper to convert a categorical variable into the first eigenvector of the dummy-expanded matrix.
static class	FrameUtils.CategoricalLabelEncoder
static class	FrameUtils.CategoricalOneHotEncoder
static class	FrameUtils.ExpandCatBetaConstraints
static class	FrameUtils.ExportTaskDriver
static class	FrameUtils.MissingInserter Helper to insert missing values into a Frame
static class	FrameUtils.Vec2ArryTsk
static class	FrameUtils.Vecs2ArryTsk
static class	FrameUtils.WeightedMean

Field Summary

Fields

Modifier and Type	Field and Description
static int	MAX_VEC_NUM_ROWS_FOR_ARRAY_EXPORT

Constructor Summary

Constructors

Constructor and Description
FrameUtils()

Method Summary

Modifier and Type	Method and Description
static double[][]	asDoubles(Frame frame)
static double[]	asDoubles(Vec v)
static int[]	asInts(Vec v)
static Frame	categoricalEncoder(Frame dataset, java.lang.String[] skipCols, Model.Parameters.CategoricalEncodingScheme scheme, ToEigenVec tev, int maxLevels)
static ChunkSummary	chunkSummary(Frame fr) Compute a chunk summary (how many chunks of each type, relative size, total size)
static void	cleanUp(IcedHashMap<Key,java.lang.String> toDelete)
static void	delete(Lockable... frs)
static Key	eagerLoadFromHTTP(java.lang.String path)
static Key<?>	eagerLoadFromURL(java.lang.String sourceId, java.net.URL url)
static Frame	encodeBetaConstraints(Key key, java.lang.String[] coefNames, java.lang.String[] coefOriginalNames, Frame betaConstraints)
static Chunk[]	extractChunks(Frame fr, int chunkId, boolean runLocal)
static Key[]	generateNumKeys(Key mk, int num) Generate given numbers of keys by suffixing key by given numbered suffix.
static Key[]	generateNumKeys(Key mk, int num, java.lang.String delim)
static void	labelRows(Frame frame, java.lang.String labelColumnName) Labels frame's rows with a sequence starting with 1 & sending with total number of rows in the frame.
static Frame	parseFrame(Key okey, java.io.File... files) Parse given file(s) into the form of single frame represented by the given key.
static Frame	parseFrame(Key okey, ParseSetup parseSetup, java.net.URI... uris)
static Frame	parseFrame(Key okey, java.net.URI... uris) Parse given set of URIs and produce a frame's key representing output.
static void	printTopCategoricalLevels(Frame fr, boolean warn, int topK)
static void	shrinkDomainsToObservedSubset(Frame frameToModifyInPlace) reduce the domains of all categorical columns to the actually observed subset
static double	sparseRatio(Chunk[] chks) compute fraction of sparse chunks in this array.
static double	sparseRatio(Frame fr)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_VEC_NUM_ROWS_FOR_ARRAY_EXPORT

public static final int MAX_VEC_NUM_ROWS_FOR_ARRAY_EXPORT

See Also:

Constant Field Values

Constructor Detail

FrameUtils

public FrameUtils()

Method Detail

parseFrame

public static Frame parseFrame(Key okey,
                               java.io.File... files)
                        throws java.io.IOException

Parse given file(s) into the form of single frame represented by the given key.

Parameters:

okey - destination key for parsed frame

files - files to parse

Returns:

a new frame

Throws:

java.io.IOException

parseFrame

public static Frame parseFrame(Key okey,
                               java.net.URI... uris)
                        throws java.io.IOException

Parse given set of URIs and produce a frame's key representing output.

Parameters:

okey - key for ouput frame. Can be null

uris - array of URI (file://, hdfs://, s3n://, s3a://, s3://, http://, https:// ...) to parse

Returns:

a frame which is saved into DKV under okey

Throws:

java.io.IOException - in case of parse error.

eagerLoadFromHTTP

public static Key eagerLoadFromHTTP(java.lang.String path)
                             throws java.io.IOException

Throws:

java.io.IOException

eagerLoadFromURL

public static Key<?> eagerLoadFromURL(java.lang.String sourceId,
                                      java.net.URL url)
                               throws java.io.IOException

Throws:

java.io.IOException

parseFrame

public static Frame parseFrame(Key okey,
                               ParseSetup parseSetup,
                               java.net.URI... uris)
                        throws java.io.IOException

Throws:

java.io.IOException

categoricalEncoder

public static Frame categoricalEncoder(Frame dataset,
                                       java.lang.String[] skipCols,
                                       Model.Parameters.CategoricalEncodingScheme scheme,
                                       ToEigenVec tev,
                                       int maxLevels)

printTopCategoricalLevels

public static void printTopCategoricalLevels(Frame fr,
                                             boolean warn,
                                             int topK)

asDoubles

public static double[] asDoubles(Vec v)

asDoubles

public static double[][] asDoubles(Frame frame)

asInts

public static int[] asInts(Vec v)

chunkSummary

public static ChunkSummary chunkSummary(Frame fr)

Compute a chunk summary (how many chunks of each type, relative size, total size)

Parameters:

fr -

Returns:

chunk summary

generateNumKeys

public static Key[] generateNumKeys(Key mk,
                                    int num)

Generate given numbers of keys by suffixing key by given numbered suffix.

generateNumKeys

public static Key[] generateNumKeys(Key mk,
                                    int num,
                                    java.lang.String delim)

sparseRatio

public static double sparseRatio(Chunk[] chks)

compute fraction of sparse chunks in this array.

Parameters:

chks -

Returns:

sparseRatio

public static double sparseRatio(Frame fr)

cleanUp

public static void cleanUp(IcedHashMap<Key,java.lang.String> toDelete)

shrinkDomainsToObservedSubset

public static void shrinkDomainsToObservedSubset(Frame frameToModifyInPlace)

reduce the domains of all categorical columns to the actually observed subset

Parameters:

frameToModifyInPlace -

delete

public static void delete(Lockable... frs)

labelRows

public static void labelRows(Frame frame,
                             java.lang.String labelColumnName)

Labels frame's rows with a sequence starting with 1 & sending with total number of rows in the frame. A vector is added to the frame given, no data are duplicated.

Parameters:

frame - Frame to label

labelColumnName - Name of the label column

encodeBetaConstraints

public static Frame encodeBetaConstraints(Key key,
                                          java.lang.String[] coefNames,
                                          java.lang.String[] coefOriginalNames,
                                          Frame betaConstraints)

extractChunks

public static Chunk[] extractChunks(Frame fr,
                                    int chunkId,
                                    boolean runLocal)