hex

Class DataInfo

java.lang.Object

water.Iced<T>

water.Keyed<DataInfo>

hex.DataInfo

All Implemented Interfaces:

java.io.Externalizable, java.io.Serializable, java.lang.Cloneable, water.Freezable<DataInfo>

public class DataInfo
extends water.Keyed<DataInfo>

Created by tomasnykodym on 1/29/15. Provides higher level interface for accessing data row-wise. Performs on the fly auto-expansion of categorical variables (to 1 hot encoding) and standardization ( or normalize/demean/descale/none) of predictors and response. Supports sparse data, sparse columns can be transformed to sparse rows on the fly with some (significant) memory overhead, as the data of the whole chunk(s) will be copied.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	DataInfo.Row
class	DataInfo.Rows
static class	DataInfo.TransformType

Field Summary

Fields

Modifier and Type	Field and Description
int[]	_activeCols
water.fvec.Frame	_adaptedFrame
int[][]	_catLvls
boolean[]	_catMissing
int[]	_catOffsets
int	_cats
java.lang.String[]	_coefNames
boolean	_fold
boolean	_imputeMissing
java.lang.String[]	_interactionColumns
hex.Model.InteractionPair[]	_interactions
int[]	_interactionVecs
boolean	_intercept
int[][]	_intLvls
double[]	_normMul
double[]	_normRespMul
double[]	_normRespSub
double[]	_normSub
double[]	_numMeans
int[]	_numOffsets
int	_nums
boolean	_offset
int	_outpus
int[]	_permutation
DataInfo.TransformType	_predictor_transform
DataInfo.TransformType	_response_transform
int	_responses
boolean	_skipMissing
boolean	_useAllFactorLevels
boolean	_valid
boolean	_weights

Fields inherited from class water.Keyed

_key

Constructor Summary

Constructors

Constructor and Description
DataInfo(water.fvec.Frame train, water.fvec.Frame valid, boolean useAllFactorLevels, DataInfo.TransformType predictor_transform, boolean skipMissing, boolean imputeMissing, boolean missingBucket)
DataInfo(water.fvec.Frame train, water.fvec.Frame valid, int nResponses, boolean useAllFactorLevels, DataInfo.TransformType predictor_transform, DataInfo.TransformType response_transform, boolean skipMissing, boolean imputeMissing, boolean missingBucket, boolean weight, boolean offset, boolean fold)
DataInfo(water.fvec.Frame train, water.fvec.Frame valid, int nResponses, boolean useAllFactorLevels, DataInfo.TransformType predictor_transform, DataInfo.TransformType response_transform, boolean skipMissing, boolean imputeMissing, boolean missingBucket, boolean weight, boolean offset, boolean fold, boolean intercept)
DataInfo(water.fvec.Frame train, water.fvec.Frame valid, int nResponses, boolean useAllFactorLevels, DataInfo.TransformType predictor_transform, DataInfo.TransformType response_transform, boolean skipMissing, boolean imputeMissing, boolean missingBucket, boolean weight, boolean offset, boolean fold, java.lang.String[] interactions) The train/valid Frame instances are sorted by categorical (themselves sorted by cardinality greatest to least) with all numerical columns following.

Method Summary

Methods

Modifier and Type	Method and Description
int[]	activeCols()
void	addOutput(java.lang.String name, water.fvec.Vec v)
void	addResponse(java.lang.String[] names, water.fvec.Vec[] vecs)
int[]	catNAFill()
int	catNAFill(int cid)
protected long	checksum_impl()
java.lang.String[]	coefNames()
double[]	denormalizeBeta(double[] beta)
void	dropInteractions()
void	dropWeights()
DataInfo.Row	extractDenseRow(water.fvec.Chunk[] chunks, int rid, DataInfo.Row row)
DataInfo.Row[]	extractSparseRows(water.fvec.Chunk[] chunks) Extract (sparse) rows from given chunks.
DataInfo	filterExpandedColumns(int[] cols) Filter the _adaptedFrame so that it contains only the Vecs referenced by the cols parameter.
int	foldChunkId()
protected int[]	fullCatOffsets()
int	fullN() Get the fully expanded number of predictor columns.
int	getCategoricalId(int cid, double val)
int	getCategoricalId(int cid, int val) Get the offset into the expanded categorical
int	getCategoricalIdFromInteraction(int cid, int val)
int	getInteractionOffset(water.fvec.Chunk[] chunks, int cid, int rid)
water.fvec.Vec	getOffsetVec()
water.fvec.Vec	getOutputVec(int i)
water.fvec.Vec	getWeightsVec()
static int	imputeCat(water.fvec.Vec v)
static int	imputeCat(water.fvec.Vec v, boolean useAllFactorLevels)
boolean	isInteractionVec(int colid)
int	largestCat()
int[]	mapNames(java.lang.String[] names)
DataInfo.Row	newDenseRow()
DataInfo.Row	newDenseRow(double[] numVals, long start)
int	nextNumericIdx(int currentColIdx) Get the next expanded number-column index.
int	numCats()
int	numNums()
int	numStart()
int	offsetChunkId()
int	outputChunkId()
int	outputChunkId(int n)
int	responseChunkId(int n)
DataInfo.Rows	rows(water.fvec.Chunk[] chks)
DataInfo.Rows	rows(water.fvec.Chunk[] chks, boolean sparse)
DataInfo	scoringInfo(java.lang.String[] names, water.fvec.Frame adaptFrame)
void	setCatNAFill(int[] catNAFill)
void	setPredictorTransform(DataInfo.TransformType t)
void	setResponse(java.lang.String name, water.fvec.Vec v)
void	setResponse(java.lang.String name, water.fvec.Vec v, int n)
void	setResponseTransform(DataInfo.TransformType t)
water.fvec.Vec	setWeights(java.lang.String name, water.fvec.Vec vec)
void	unScaleNumericals(double[] in, double[] out) Undo the standardization/normalization of numerical columns
void	updateWeightedSigmaAndMean(double[] sigmas, double[] mean)
void	updateWeightedSigmaAndMeanForResponse(double[] sigmas, double[] mean)
DataInfo	validDinfo(water.fvec.Frame valid)
int	weightChunkId()

Methods inherited from class water.Keyed

checksum, makeSchema, readAll_impl, readAll, remove_impl, remove, remove, remove, remove, writeAll_impl, writeAll

Methods inherited from class water.Iced

asBytes, clone, copyOver, frozenType, read, readExternal, readJSON, reloadFromBytes, toJsonString, write, writeExternal, writeJSON

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

_activeCols

public int[] _activeCols

_adaptedFrame

public water.fvec.Frame _adaptedFrame

_responses

public int _responses

_outpus

public int _outpus

_predictor_transform

public DataInfo.TransformType _predictor_transform

_response_transform

public DataInfo.TransformType _response_transform

_useAllFactorLevels

public boolean _useAllFactorLevels

_nums

public int _nums

_cats

public int _cats

_catOffsets

public int[] _catOffsets

_catMissing

public boolean[] _catMissing

_permutation

public int[] _permutation

_normMul

public double[] _normMul

_normSub

public double[] _normSub

_normRespMul

public double[] _normRespMul

_normRespSub

public double[] _normRespSub

_numMeans

public double[] _numMeans

_intercept

public boolean _intercept

_offset

public boolean _offset

_weights

public boolean _weights

_fold

public boolean _fold

_interactions

public hex.Model.InteractionPair[] _interactions

_interactionColumns

public java.lang.String[] _interactionColumns

_interactionVecs

public int[] _interactionVecs

_numOffsets

public int[] _numOffsets

_skipMissing

public final boolean _skipMissing

_imputeMissing

public final boolean _imputeMissing

_valid

public boolean _valid

_catLvls

public final int[][] _catLvls

_intLvls

public final int[][] _intLvls

_coefNames

public java.lang.String[] _coefNames

Constructor Detail

DataInfo

public DataInfo(water.fvec.Frame train,
        water.fvec.Frame valid,
        boolean useAllFactorLevels,
        DataInfo.TransformType predictor_transform,
        boolean skipMissing,
        boolean imputeMissing,
        boolean missingBucket)

DataInfo

public DataInfo(water.fvec.Frame train,
        water.fvec.Frame valid,
        int nResponses,
        boolean useAllFactorLevels,
        DataInfo.TransformType predictor_transform,
        DataInfo.TransformType response_transform,
        boolean skipMissing,
        boolean imputeMissing,
        boolean missingBucket,
        boolean weight,
        boolean offset,
        boolean fold)

DataInfo

public DataInfo(water.fvec.Frame train,
        water.fvec.Frame valid,
        int nResponses,
        boolean useAllFactorLevels,
        DataInfo.TransformType predictor_transform,
        DataInfo.TransformType response_transform,
        boolean skipMissing,
        boolean imputeMissing,
        boolean missingBucket,
        boolean weight,
        boolean offset,
        boolean fold,
        java.lang.String[] interactions)

The train/valid Frame instances are sorted by categorical (themselves sorted by cardinality greatest to least) with all numerical columns following. The response column(s) are placed at the end. Interactions: 1. Num-Num (Note: N(0,1) * N(0,1) ~ N(0,1) ) 2. Num-Enum 3. Enum-Enum Interactions are produced on the fly and are dense (in all 3 cases). Consumers of DataInfo should not have to care how these interactions are generated. Any heuristic using the fullN value should continue functioning the same. Interactions are specified in two ways: A. As a list of pairs of column indices. B. As a list of pairs of column indices with limited enums.

DataInfo

public DataInfo(water.fvec.Frame train,
        water.fvec.Frame valid,
        int nResponses,
        boolean useAllFactorLevels,
        DataInfo.TransformType predictor_transform,
        DataInfo.TransformType response_transform,
        boolean skipMissing,
        boolean imputeMissing,
        boolean missingBucket,
        boolean weight,
        boolean offset,
        boolean fold,
        boolean intercept)

Method Detail

setWeights

public water.fvec.Vec setWeights(java.lang.String name,
                        water.fvec.Vec vec)

dropWeights

public void dropWeights()

dropInteractions

public void dropInteractions()

activeCols

public int[] activeCols()

addResponse

public void addResponse(java.lang.String[] names,
               water.fvec.Vec[] vecs)

catNAFill

public int[] catNAFill()

catNAFill

public int catNAFill(int cid)

setCatNAFill

public void setCatNAFill(int[] catNAFill)

responseChunkId

public int responseChunkId(int n)

foldChunkId

public int foldChunkId()

offsetChunkId

public int offsetChunkId()

weightChunkId

public int weightChunkId()

outputChunkId

public int outputChunkId()

outputChunkId

public int outputChunkId(int n)

addOutput

public void addOutput(java.lang.String name,
             water.fvec.Vec v)

getOutputVec

public water.fvec.Vec getOutputVec(int i)

setResponse

public void setResponse(java.lang.String name,
               water.fvec.Vec v)

setResponse

public void setResponse(java.lang.String name,
               water.fvec.Vec v,
               int n)

checksum_impl

protected long checksum_impl()

Overrides:

checksum_impl in class water.Keyed<DataInfo>

validDinfo

public DataInfo validDinfo(water.fvec.Frame valid)

denormalizeBeta

public double[] denormalizeBeta(double[] beta)

fullCatOffsets

protected int[] fullCatOffsets()

imputeCat

public static int imputeCat(water.fvec.Vec v)

imputeCat

public static int imputeCat(water.fvec.Vec v,
            boolean useAllFactorLevels)

filterExpandedColumns

public DataInfo filterExpandedColumns(int[] cols)

Filter the _adaptedFrame so that it contains only the Vecs referenced by the cols parameter.

Parameters:

cols - Array of the expanded column indices to keep.

Returns:

A DataInfo with _activeCols specifying the active columns

updateWeightedSigmaAndMean

public void updateWeightedSigmaAndMean(double[] sigmas,
                              double[] mean)

updateWeightedSigmaAndMeanForResponse

public void updateWeightedSigmaAndMeanForResponse(double[] sigmas,
                                         double[] mean)

setPredictorTransform

public void setPredictorTransform(DataInfo.TransformType t)

setResponseTransform

public void setResponseTransform(DataInfo.TransformType t)

isInteractionVec

public boolean isInteractionVec(int colid)

fullN

public final int fullN()

Get the fully expanded number of predictor columns. Note that this value does not include: response column(s) weight column offset column fold column

Returns:

expanded number of columns in the underlying frame

largestCat

public final int largestCat()

numStart

public final int numStart()

numCats

public final int numCats()

numNums

public final int numNums()

nextNumericIdx

public final int nextNumericIdx(int currentColIdx)

Get the next expanded number-column index.

coefNames

public final java.lang.String[] coefNames()

mapNames

public int[] mapNames(java.lang.String[] names)

unScaleNumericals

public final void unScaleNumericals(double[] in,
                     double[] out)

Undo the standardization/normalization of numerical columns

Parameters:

in - input values

out - output values (can be the same as input)

getCategoricalId

public final int getCategoricalId(int cid,
                   double val)

getCategoricalId

public final int getCategoricalId(int cid,
                   int val)

Get the offset into the expanded categorical

Parameters:

cid - the column id

val - the integer representation of the categorical level

Returns:

offset into the fullN set of columns

getCategoricalIdFromInteraction

public final int getCategoricalIdFromInteraction(int cid,
                                  int val)

extractDenseRow

public final DataInfo.Row extractDenseRow(water.fvec.Chunk[] chunks,
                           int rid,
                           DataInfo.Row row)

getInteractionOffset

public int getInteractionOffset(water.fvec.Chunk[] chunks,
                       int cid,
                       int rid)

getWeightsVec

public water.fvec.Vec getWeightsVec()

getOffsetVec

public water.fvec.Vec getOffsetVec()

newDenseRow

public DataInfo.Row newDenseRow()

newDenseRow

public DataInfo.Row newDenseRow(double[] numVals,
                       long start)

rows

public DataInfo.Rows rows(water.fvec.Chunk[] chks)

rows

public DataInfo.Rows rows(water.fvec.Chunk[] chks,
                 boolean sparse)

extractSparseRows

public final DataInfo.Row[] extractSparseRows(water.fvec.Chunk[] chunks)

Extract (sparse) rows from given chunks. Note: 0 remains 0 - _normSub of DataInfo isn't used (mean shift during standarization is not reverted) - UNLESS offset is specified (for GLM only) Essentially turns the dataset 90 degrees.

Parameters:

chunks - - chunk of dataset

Returns:

array of sparse rows

scoringInfo

public DataInfo scoringInfo(java.lang.String[] names,
                   water.fvec.Frame adaptFrame)