water.fvec

Class Vec

java.lang.Object

water.Iced<T>

water.Keyed<Vec>

water.fvec.Vec

All Implemented Interfaces:

java.io.Externalizable, java.io.Serializable, java.lang.Cloneable, Freezable<Vec>

Direct Known Subclasses:

AppendableVec, ByteVec, CategoricalWrappedVec, InteractionWrappedVec, SubsetVec, TransformWrappedVec

public class Vec
extends Keyed<Vec>

A distributed vector/array/column of uniform data.

A distributed vector has a count of elements, an element-to-chunk mapping, a Java-like type (mostly determines rounding on store and display), and functions to directly load elements without further indirections. The data is compressed, or backed by disk or both.

A Vec is a collection of Chunks, each of which holds between 1,000 and 1,000,000 elements. Operations on a Chunk are intended to be single-threaded; operations on a Vec are intended to be parallel and distributed on Chunk granularities, with each Chunk being manipulated by a separate CPU. The standard Map/Reduce (MRTask) paradigm handles parallel and distributed Chunk access well.

Individual elements can be directly accessed like a (very large and distributed) array - however this is not the fastest way to access the data. Direct access from Chunks is faster, avoiding several layers of indirection. In particular accessing a random row from the Vec will force the containing Chunk data to be cached locally (and network traffic to bring it local); accessing all rows from a single machine will force all the Big Data to be pulled local typically resulting in swapping and very poor performance. The main API is provided for ease of small-data manipulations and is fairly slow for writing; when touching ALL the data you are much better off using e.g. MRTask.

The main API is at(long), set(long, long), and isNA(long):

Returns	Call	Missing?	Notes
double	at(long)	NaN
long	at8(long)	throws
long	at16l(long)	throws	Low half of 128-bit UUID
long	at16h(long)	throws	High half of 128-bit UUID
BufferedString	atStr(water.parser.BufferedString, long)	null	Updates BufferedString in-place and returns it for flow-coding
boolean	isNA(long)	true
	set(long,double)	NaN
	set(long,float)	NaN	Limited precision takes less memory
	set(long,long)	Cannot set
	set(long,String)	null	Convenience wrapper for String
	setNA(long)

Example manipulating some individual elements:

    double r1 = vec.at(0x123456789L);  // Access element 0x1234567889 as a double
    double r2 = vec.at(-1);            // Throws AIOOBE
    long   r3 = vec.at8_abs(1);        // Element #1, as a long
    vec.set(2,r1+r3);                  // Set element #2, as a double
  

Vecs have a loosely enforced type: one of numeric, UUID or String. Numeric types are further broken down into integral (long) and real (double) types. The categorical type is an integral type, with a String mapping side-array. Most of the math algorithms will treat categoricals as small dense integers, and most categorical printouts will use the String mapping. Time is another special integral type: it is represented as milliseconds since the unix epoch, and is mostly treated as an integral type when doing math but it has special time-based printout formatting. All types support the notion of a missing element; for real types this is always NaN. It is an error to attempt to fetch a missing integral type, and isNA(long) must be called first. Integral types are losslessly compressed. Real types may lose 1 or 2 ULPS due to compression.

Reading elements as doubles, or checking for an element missing is always safe. Reading a missing integral type throws an exception, since there is no NaN equivalent in the integer domain. Writing to elements may throw if the backing data is read-only (file backed), and otherwise is fully supported.

Note this dangerous scenario: loading a missing value as a double, and setting it as a long:

   set(row,(long)at(row)); // Danger!

The cast from a Double.NaN to a long produces a zero! This code will silently replace a missing value with a zero.

Vecs have a lazily computed RollupStats object and Key. The RollupStats give fast access to the common metrics: min(), max(), mean(), sigma(), the count of missing elements (naCnt()) and non-zeros (nzCnt()), amongst other stats. They are cleared if the Vec is modified and lazily recomputed after the modified Vec is closed. Clearing the RollupStats cache is fairly expensive for individual set(long, long) calls but is easy to amortize over a large count of writes; i.e., batch writing is efficient. This is normally handled by the MRTask framework; the Vec.Writer framework allows single-threaded efficient batch writing for smaller Vecs.

Example usage of common stats:

    double mean = vec.mean();  // Vec's mean; first touch computes and caches rollups
    double min  = vec.min();   // Smallest element; already computed
    double max  = vec.max();   // Largest element; already computed
    double sigma= vec.sigma(); // Standard deviation; already computed
  

Example: Impute (replace) missing values with the mean. Note that the use of vec.mean() in the constructor uses (and computes) the general RollupStats before the MRTask starts. Setting a value in the Chunk clears the RollupStats (since setting any value but the mean will change the mean); they will be recomputed at the next use after the MRTask.

    new MRTask{} { final double _mean = vec.mean();
      public void map( Chunk chk ) {
        for( int row=0; row < chk._len; row++ )
          if( chk.isNA(row) ) chk.set(row,_mean);
      }
    }.doAll(vec);
  

Vecs have a Vec.VectorGroup. Vecs in the same VectorGroup have the same Chunk and row alignment - that is, Chunks with the same index are homed to the same Node and have the same count of rows-per-Chunk. Frames are only composed of Vecs of the same VectorGroup (or very small Vecs) guaranteeing that all elements of each row are homed to the same Node and set of Chunks - such that a simple for loop over a set of Chunks all operates locally. See the example in the Chunk class.

It is common and cheap to make new Vecs in the same VectorGroup as an existing Vec and initialized to e.g. zero. Such Vecs are often used as temps, and usually immediately set to interest values in a later MRTask pass.

Example creation of temp Vecs:

    Vec tmp0 = vec.makeZero();         // New Vec with same VectorGroup and layout as vec, filled with zero
    Vec tmp1 = vec.makeCon(mean);      // Filled with 'mean'
    assert tmp1.at(0x123456789)==mean; // All elements filled with 'mean'
    for( int i=0; i<100; i++ )         // A little math on the first 100 elements
      tmp0.set(i,tmp1.at(i)+17);       // ...set into the tmp0 vec
  

Vec Keys have a special layout (enforced by the various Vec constructors) so there is a direct Key mapping from a Vec to a numbered Chunk and vice-versa. This mapping is crucially used in all sorts of places, basically representing a global naming scheme across a Vec and the Chunks that make it up. The basic layout created by newKey():

              byte:    0      1   2 3 4 5  6 7 8 9  10+
  Vec   Key layout: Key.VEC  -1   vec#grp    -1     normal Key bytes; often e.g. a function of original file name
  Chunk Key layout: Key.CHK  -1   vec#grp  chunk#   normal Key bytes; often e.g. a function of original file name
  RollupStats Key : Key.CHK  -1   vec#grp    -2     normal Key bytes; often e.g. a function of original file name
  Group Key layout: Key.GRP  -1     -1       -1     normal Key bytes; often e.g. a function of original file name
  ESPC  Key layout: Key.GRP  -1     -1       -2     normal Key bytes; often e.g. a function of original file name
 

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Vec.ESPC
class	Vec.Reader A more efficient way to read randomly to a Vec - still single-threaded, but much faster than Vec.at(i).
static class	Vec.VectorGroup Class representing the group of vectors.
class	Vec.Writer A more efficient way to write randomly to a Vec - still single-threaded, still slow, but much faster than Vec.set().

Field Summary

Fields

Modifier and Type	Field and Description
int	_rowLayout Element-start per chunk, i.e.
static boolean	DO_HISTOGRAMS
static int	KEY_PREFIX_LEN Internally used to help build Vec and Chunk Keys; public to help PersistNFS build file mappings.
static double[]	PERCENTILES Default percentiles for approximate (single-pass) quantile computation (histogram-based).
static byte	T_BAD
static byte	T_CAT
static byte	T_NUM
static byte	T_STR
static byte	T_TIME
static byte	T_UUID
static java.lang.String[]	TYPE_STR

Fields inherited from class water.Keyed

_key

Constructor Summary

Constructors

Constructor and Description
Vec(Key<Vec> key, int rowLayout) Build a numeric-type Vec; the caller understands Chunk layout (via the espc array).
Vec(Key<Vec> key, int rowLayout, java.lang.String[] domain, byte type) Main default constructor; the caller understands Chunk layout (via the espc array), plus categorical/factor the domain (or null for non-categoricals), and the Vec type.

Method Summary

Methods

Modifier and Type	Method and Description
CategoricalWrappedVec	adaptTo(java.lang.String[] domain) Make a Vec adapting this cal vector to the 'to' categorical Vec.
Vec	align(Vec vec) Always makes a copy of the given vector which shares the same group as this Vec.
double	at(long i) Fetch element the slow way, as a double, or Double.NaN is missing.
long	at16h(long i) Fetch element the slow way, as the high half of a UUID.
long	at16l(long i) Fetch element the slow way, as the low half of a UUID.
long	at8(long i) Fetch element the slow way, as a long.
BufferedString	atStr(BufferedString bStr, long i) Fetch element the slow way, as a BufferedString or null if missing.
double	base() The base for a simple and cheap histogram of the Vec, useful for getting a broad overview of the data.
long[]	bins() A simple and cheap histogram of the Vec, useful for getting a broad overview of the data.
long	byteSize() Size of compressed vector data.
int	cardinality() Returns cardinality for categorical domain or -1 for other types.
protected long	checksum_impl() A high-quality 64-bit checksum of the Vec's content, useful for establishing dataset identity.
Chunk	chunkForChunkIdx(int cidx) The Chunk for a chunk#.
Chunk	chunkForRow(long i) The Chunk for a row#.
Value	chunkIdx(int cidx) Get a Chunk's Value by index.
Key	chunkKey(int cidx) Get a Chunk Key from a chunk-index.
static Key	chunkKey(Key veckey, int cidx) Get a Chunk Key from a chunk-index and a Vec Key, without needing the actual Vec object.
void	copyMeta(Vec src, Futures fs)
Vec	doCopy()
java.lang.String[]	domain() Returns the categorical toString mapping array, or null if not an categorical column.
int	elem2ChunkIdx(long i) Convert a row# to a chunk#.
boolean	equals(java.lang.Object o) True if two Vecs are equal.
long[]	espc()
java.lang.String	factor(long i) Returns the ith factor for this categorical column.
java.lang.String	get_type_str()
byte	get_type() Get the column type.
static Key	getVecKey(Key chk_key) Get a Vec Key from Chunk Key, without loading the Chunk.
Vec.VectorGroup	group() Get the group this vector belongs to.
int	hashCode() Vec's hashcode, which is just the Vec Key hashcode.
boolean	isBad() True if the column contains only NAs
boolean	isBinary()
boolean	isCategorical() True if this is an categorical column.
boolean	isConst() True if the column contains only a constant value and it is not full of NAs
boolean	isInt() isInt is a property of numeric Vecs and not a type; this property can be changed by assigning non-integer values into the Vec (or restored by overwriting non-integer values with integers).
boolean	isNA(long row) Fetch the missing-status the slow way.
boolean	isNumeric() True if this is a numeric column, excluding categorical and time types.
boolean	isString() True if this is a String column.
boolean	isTime() True if this is a time column.
boolean	isUUID() True if this is a UUID column.
long[]	lazy_bins() Optimistically return the histogram bins, or null if not computed
long	length() Number of elements in the vector; returned as a long instead of an int because Vecs support more than 2^32 elements.
Vec	makeCon(double d) Make a new vector with the same size and data layout as the current one, and initialized to the given constant value.
static Vec	makeCon(double x, long len) Make a new constant vector with the given row count, and redistribute the data evenly around the cluster.
static Vec	makeCon(double x, long len, boolean redistribute) Make a new constant vector with the given row count.
static Vec	makeCon(double x, long len, int log_rows_per_chunk) Make a new constant vector with the given row count, and redistribute the data evenly around the cluster.
static Vec	makeCon(double x, long len, int log_rows_per_chunk, boolean redistribute) Make a new constant vector with the given row count.
static Vec	makeCon(Key<Vec> k, double... rows) A Vec from an array of doubles
static Vec	makeCon(long totSize, long len) Make a new constant vector with minimal number of chunks.
static Vec	makeCon(long l, java.lang.String[] domain, Vec.VectorGroup group, int rowLayout)
static Vec[]	makeCons(double x, long len, int n)
Vec[]	makeCons(int n, long l, java.lang.String[][] domains, byte[] types)
Vec	makeCopy() A new vector which is a copy of this one.
Vec	makeCopy(java.lang.String[] domain) A new vector which is a copy of this one.
Vec	makeCopy(java.lang.String[] domain, byte type)
Vec[]	makeDoubles(int n, double[] values)
Vec	makeRand(long seed) Make a new vector initialized to random numbers with the given seed
static Vec	makeRepSeq(long len, long repeat) Make a new vector initialized to increasing integers mod repeat.
static Vec	makeSeq(long len, boolean redistribute) Make a new vector initialized to increasing integers, starting with 1.
static Vec	makeSeq(long min, long len) Make a new vector initialized to increasing integers, starting with `min`.
static Vec	makeSeq(long min, long len, boolean redistribute) Make a new vector initialized to increasing integers, starting with `min`.
static Vec	makeVec(double[] vals, Key<Vec> vecKey)
static Vec	makeVec(double[] vals, java.lang.String[] domain, Key<Vec> vecKey)
static Vec	makeVec(long[] vals, java.lang.String[] domain, Key<Vec> vecKey)
Vec	makeZero() Make a new vector with the same size and data layout as the current one, and initialized to zero.
static Vec	makeZero(long len) Make a new zero-filled vector with the given row count.
static Vec	makeZero(long len, boolean redistribute) Make a new zero-filled vec
Vec	makeZero(java.lang.String[] domain) A new vector with the same size and data layout as the current one, and initialized to zero, with the given categorical domain.
Vec[]	makeZeros(int n)
Vec[]	makeZeros(int n, java.lang.String[][] domain, byte[] types)
double	max() Vec's maximum value
double[]	maxs() Vec's 5 largest values
double	mean() Vecs's mean
double	min() Vec's minimum value
double[]	mins() Vec's 5 smallest values
int	mode() Vecs's mode
long	naCnt() Count of missing elements
int	nChunks() Number of chunks, returned as an int - Chunk count is limited by the max size of a Java long[].
static Key<Vec>	newKey() Make a new random Key that fits the requirements for a Vec key.
long	ninfs() Count of negative infinities
long	nzCnt() Count of non-zero elements
Vec.Writer	open() Create a writer for bulk serial writes into this Vec.
double[]	pctiles() A simple and cheap percentiles of the Vec, useful for getting a broad overview of the data.
long	pinfs() Count of positive infinities
Futures	postWrite(Futures fs) Stop writing into this Vec.
void	preWriting() Begin writing into this Vec.
protected Keyed	readAll_impl(AutoBuffer ab, Futures fs)
Futures	remove_impl(Futures fs) Remove associated Keys when this guy removes.
Key	rollupStatsKey()
void	set(long i, double d) Write element the slow way, as a double.
void	set(long i, float f) Write element the slow way, as a float.
void	set(long i, long l) Write element the slow way, as a long.
void	set(long i, java.lang.String str) Write element the slow way, as a String.
void	setBad()
void	setDomain(java.lang.String[] domain) Set the categorical/factor names.
double	sigma() Vecs's standard deviation
double	sparseRatio()
void	startRollupStats(Futures fs)
void	startRollupStats(Futures fs, boolean doHisto) Check if we have local cached copy of basic Vec stats (including histogram if requested) and if not start task to compute and fetch them; useful when launching a bunch of them in parallel to avoid single threaded execution later (e.g.
double	stride() The stride for a a simple and cheap histogram of the Vec, useful for getting a broad overview of the data.
Vec	toCategoricalVec() Convenience method for converting to a categorical vector.
Vec	toNumericVec() Convenience method for converting to a numeric vector.
java.lang.String	toString() Pretty print the Vec: [#elems, min/mean/max]{chunks,...}
Vec	toStringVec() Convenience method for converting to a string vector.
protected AutoBuffer	writeAll_impl(AutoBuffer ab) Write out K/V pairs

Methods inherited from class water.Keyed

checksum, makeSchema, readAll, remove, remove, remove, remove, writeAll

Methods inherited from class water.Iced

asBytes, clone, copyOver, frozenType, read, readExternal, readJSON, reloadFromBytes, toJsonString, write, writeExternal, writeJSON

Methods inherited from class java.lang.Object

finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

_rowLayout

public int _rowLayout

Element-start per chunk, i.e. the row layout. Defined in the VectorGroup. This field is dead/ignored in subclasses that are guaranteed to have fixed-sized chunks such as file-backed Vecs.

T_BAD

public static final byte T_BAD

See Also:

Constant Field Values

T_UUID

public static final byte T_UUID

See Also:

Constant Field Values

T_STR

public static final byte T_STR

See Also:

Constant Field Values

T_NUM

public static final byte T_NUM

See Also:

Constant Field Values

T_CAT

public static final byte T_CAT

See Also:

Constant Field Values

T_TIME

public static final byte T_TIME

See Also:

Constant Field Values

TYPE_STR

public static final java.lang.String[] TYPE_STR

DO_HISTOGRAMS

public static final boolean DO_HISTOGRAMS

See Also:

Constant Field Values

PERCENTILES

public static final double[] PERCENTILES

Default percentiles for approximate (single-pass) quantile computation (histogram-based).

KEY_PREFIX_LEN

public static final int KEY_PREFIX_LEN

Internally used to help build Vec and Chunk Keys; public to help PersistNFS build file mappings. Not intended as a public field.

See Also:

Constant Field Values

Constructor Detail

Vec

public Vec(Key<Vec> key,
   int rowLayout)

Build a numeric-type Vec; the caller understands Chunk layout (via the espc array).

Vec

public Vec(Key<Vec> key,
   int rowLayout,
   java.lang.String[] domain,
   byte type)

Main default constructor; the caller understands Chunk layout (via the espc array), plus categorical/factor the domain (or null for non-categoricals), and the Vec type.

Method Detail

domain

public java.lang.String[] domain()

Returns the categorical toString mapping array, or null if not an categorical column. Not a defensive clone (to expensive to clone; coding error to change the contents).

Returns:

the categorical / factor / categorical mapping array, or null if not a categorical column

factor

public final java.lang.String factor(long i)

Returns the ith factor for this categorical column.

Returns:

The ith factor

setDomain

public final void setDomain(java.lang.String[] domain)

Set the categorical/factor names. No range-checking on the actual underlying numeric domain; user is responsible for maintaining a mapping which is coherent with the Vec contents.

cardinality

public final int cardinality()

Returns cardinality for categorical domain or -1 for other types.

isCategorical

public final boolean isCategorical()

True if this is an categorical column. All categorical columns are also isInt(), but not vice-versa.

Returns:

true if this is an categorical column.

sparseRatio

public final double sparseRatio()

isUUID

public final boolean isUUID()

True if this is a UUID column.

Returns:

true if this is a UUID column.

isString

public final boolean isString()

True if this is a String column.

Returns:

true if this is a String column.

isNumeric

public final boolean isNumeric()

True if this is a numeric column, excluding categorical and time types.

Returns:

true if this is a numeric column, excluding categorical and time types

isTime

public final boolean isTime()

True if this is a time column. All time columns are also isInt(), but not vice-versa.

Returns:

true if this is a time column.

espc

public long[] espc()

length

public long length()

Number of elements in the vector; returned as a long instead of an int because Vecs support more than 2^32 elements. Overridden by subclasses that compute length in an alternative way, such as file-backed Vecs.

Returns:

Number of elements in the vector

nChunks

public int nChunks()

Number of chunks, returned as an int - Chunk count is limited by the max size of a Java long[]. Overridden by subclasses that compute chunks in an alternative way, such as file-backed Vecs.

Returns:

Number of chunks

setBad

public void setBad()

get_type

public byte get_type()

Get the column type.

get_type_str

public java.lang.String get_type_str()

isBinary

public boolean isBinary()

copyMeta

public void copyMeta(Vec src,
            Futures fs)

makeZero

public static Vec makeZero(long len,
           boolean redistribute)

Make a new zero-filled vec

makeZero

public static Vec makeZero(long len)

Make a new zero-filled vector with the given row count.

Returns:

New zero-filled vector with the given row count.

makeCon

public static Vec makeCon(double x,
          long len)

Make a new constant vector with the given row count, and redistribute the data evenly around the cluster.

Parameters:

x - The value with which to fill the Vec.

len - Number of rows.

Returns:

New cosntant vector with the given len.

makeCon

public static Vec makeCon(double x,
          long len,
          boolean redistribute)

Make a new constant vector with the given row count.

Returns:

New constant vector with the given row count.

makeCon

public static Vec makeCon(double x,
          long len,
          int log_rows_per_chunk)

Make a new constant vector with the given row count, and redistribute the data evenly around the cluster.

Returns:

New constant vector with the given row count.

makeCon

public static Vec makeCon(long totSize,
          long len)

Make a new constant vector with minimal number of chunks. Used for importing SQL tables.

Returns:

New constant vector with the given row count.

makeCon

public static Vec makeCon(double x,
          long len,
          int log_rows_per_chunk,
          boolean redistribute)

Make a new constant vector with the given row count.

Returns:

New constant vector with the given row count.

makeDoubles

public Vec[] makeDoubles(int n,
                double[] values)

makeZero

public Vec makeZero()

Make a new vector with the same size and data layout as the current one, and initialized to zero.

Returns:

A new vector with the same size and data layout as the current one, and initialized to zero.

makeZero

public Vec makeZero(java.lang.String[] domain)

A new vector with the same size and data layout as the current one, and initialized to zero, with the given categorical domain.

Returns:

A new vector with the same size and data layout as the current one, and initialized to zero, with the given categorical domain.

makeCopy

public Vec makeCopy()

A new vector which is a copy of this one.

Returns:

a copy of the vector.

makeCopy

public Vec makeCopy(java.lang.String[] domain)

A new vector which is a copy of this one.

Returns:

a copy of the vector.

makeCopy

public Vec makeCopy(java.lang.String[] domain,
           byte type)

doCopy

public Vec doCopy()

makeCon

public static Vec makeCon(long l,
          java.lang.String[] domain,
          Vec.VectorGroup group,
          int rowLayout)

makeVec

public static Vec makeVec(double[] vals,
          Key<Vec> vecKey)

makeVec

public static Vec makeVec(double[] vals,
          java.lang.String[] domain,
          Key<Vec> vecKey)

makeVec

public static Vec makeVec(long[] vals,
          java.lang.String[] domain,
          Key<Vec> vecKey)

makeCons

public static Vec[] makeCons(double x,
             long len,
             int n)

makeCon

public Vec makeCon(double d)

Make a new vector with the same size and data layout as the current one, and initialized to the given constant value.

Returns:

A new vector with the same size and data layout as the current one, and initialized to the given constant value.

makeZeros

public Vec[] makeZeros(int n)

makeZeros

public Vec[] makeZeros(int n,
              java.lang.String[][] domain,
              byte[] types)

makeCons

public Vec[] makeCons(int n,
             long l,
             java.lang.String[][] domains,
             byte[] types)

makeCon

public static Vec makeCon(Key<Vec> k,
          double... rows)

A Vec from an array of doubles

Parameters:

rows - Data

Returns:

The Vec

makeSeq

public static Vec makeSeq(long len,
          boolean redistribute)

Make a new vector initialized to increasing integers, starting with 1.

Returns:

A new vector initialized to increasing integers, starting with 1.

makeSeq

public static Vec makeSeq(long min,
          long len)

Make a new vector initialized to increasing integers, starting with `min`.

Returns:

A new vector initialized to increasing integers, starting with `min`.

makeSeq

public static Vec makeSeq(long min,
          long len,
          boolean redistribute)

Make a new vector initialized to increasing integers, starting with `min`.

Returns:

A new vector initialized to increasing integers, starting with `min`.

makeRepSeq

public static Vec makeRepSeq(long len,
             long repeat)

Make a new vector initialized to increasing integers mod repeat.

Returns:

A new vector initialized to increasing integers mod repeat.

makeRand

public Vec makeRand(long seed)

Make a new vector initialized to random numbers with the given seed

min

public double min()

Vec's minimum value

Returns:

Vec's minimum value

mins

public double[] mins()

Vec's 5 smallest values

Returns:

Vec's 5 smallest values

max

public double max()

Vec's maximum value

Returns:

Vec's maximum value

maxs

public double[] maxs()

Vec's 5 largest values

Returns:

Vec's 5 largeest values

isConst

public final boolean isConst()

True if the column contains only a constant value and it is not full of NAs

Returns:

True if the column is constant

isBad

public final boolean isBad()

True if the column contains only NAs

Returns:

True if the column contains only NAs

mean

public double mean()

Vecs's mean

Returns:

Vec's mean

sigma

public double sigma()

Vecs's standard deviation

Returns:

Vec's standard deviation

mode

public int mode()

Vecs's mode

Returns:

Vec's mode

naCnt

public long naCnt()

Count of missing elements

Returns:

Count of missing elements

nzCnt

public long nzCnt()

Count of non-zero elements

Returns:

Count of non-zero elements

pinfs

public long pinfs()

Count of positive infinities

Returns:

Count of positive infinities

ninfs

public long ninfs()

Count of negative infinities

Returns:

Count of negative infinities

isInt

public boolean isInt()

isInt is a property of numeric Vecs and not a type; this property can be changed by assigning non-integer values into the Vec (or restored by overwriting non-integer values with integers). This is a strong type for isCategorical() and isTime() Vecs.

Returns:

true if the Vec is all integers

byteSize

public long byteSize()

Size of compressed vector data.

bins

public long[] bins()

A simple and cheap histogram of the Vec, useful for getting a broad overview of the data. Each bin is row-counts for the bin's range. The bin's range is computed from base() and stride(). The histogram is computed on first use and cached thereafter.

Returns:

A set of histogram bins, or null for String columns

lazy_bins

public long[] lazy_bins()

Optimistically return the histogram bins, or null if not computed

Returns:

the histogram bins, or null if not computed

base

public double base()

The base for a simple and cheap histogram of the Vec, useful for getting a broad overview of the data. This returns the base of bins()[0].

Returns:

the base of bins()[0]

stride

public double stride()

The stride for a a simple and cheap histogram of the Vec, useful for getting a broad overview of the data. This returns the stride between any two bins.

Returns:

the stride between any two bins

pctiles

public double[] pctiles()

A simple and cheap percentiles of the Vec, useful for getting a broad overview of the data. The specific percentiles are take from PERCENTILES.

Returns:

A set of percentiles

startRollupStats

public void startRollupStats(Futures fs)

startRollupStats

public void startRollupStats(Futures fs,
                    boolean doHisto)

Check if we have local cached copy of basic Vec stats (including histogram if requested) and if not start task to compute and fetch them; useful when launching a bunch of them in parallel to avoid single threaded execution later (e.g. for-loop accessing min/max of every vec in a frame). Does *not* guarantee rollup stats will be cached locally when done since there may be racy changes to vec which will flush local caches.

Parameters:

fs - Futures allow to wait for this task to finish.

doHisto - Also compute histogram, requires second pass over data amd is not computed by default.

checksum_impl

protected long checksum_impl()

A high-quality 64-bit checksum of the Vec's content, useful for establishing dataset identity.

Overrides:

checksum_impl in class Keyed<Vec>

Returns:

Checksum of the Vec's content

preWriting

public void preWriting()

Begin writing into this Vec. Immediately clears all the rollup stats (min(), max(), mean(), etc) since such values are not meaningful while the Vec is being actively modified. Can be called repeatedly. Per-chunk row-counts will not be changing, just row contents.

postWrite

public Futures postWrite(Futures fs)

Stop writing into this Vec. Rollup stats will again (lazily) be computed.

elem2ChunkIdx

public int elem2ChunkIdx(long i)

Convert a row# to a chunk#. For constant-sized chunks this is a little shift-and-add math. For variable-sized chunks this is a binary search, with a sane API (JDK has an insane API). Overridden by subclasses that compute chunks in an alternative way, such as file-backed Vecs.

getVecKey

public static Key getVecKey(Key chk_key)

Get a Vec Key from Chunk Key, without loading the Chunk.

Returns:

the Vec Key for the Chunk Key

chunkKey

public Key chunkKey(int cidx)

Get a Chunk Key from a chunk-index. Basically the index-to-key map.

Returns:

Chunk Key from a chunk-index

chunkKey

public static Key chunkKey(Key veckey,
           int cidx)

Get a Chunk Key from a chunk-index and a Vec Key, without needing the actual Vec object. Basically the index-to-key map.

Returns:

Chunk Key from a chunk-index and Vec Key

rollupStatsKey

public Key rollupStatsKey()

chunkIdx

public Value chunkIdx(int cidx)

Get a Chunk's Value by index. Basically the index-to-key map, plus the DKV.get(). Warning: this pulls the data locally; using this call on every Chunk index on the same node will probably trigger an OOM!

newKey

public static Key<Vec> newKey()

Make a new random Key that fits the requirements for a Vec key.

Returns:

A new random Vec Key

group

public final Vec.VectorGroup group()

Get the group this vector belongs to. In case of a group with only one vector, the object actually does not exist in KV store. This is the ONLY place VectorGroups are fetched.

Returns:

VectorGroup this vector belongs to

chunkForChunkIdx

public Chunk chunkForChunkIdx(int cidx)

The Chunk for a chunk#. Warning: this pulls the data locally; using this call on every Chunk index on the same node will probably trigger an OOM!

Returns:

Chunk for a chunk#

chunkForRow

public final Chunk chunkForRow(long i)

The Chunk for a row#. Warning: this pulls the data locally; using this call on every Chunk index on the same node will probably trigger an OOM!

Returns:

Chunk for a row#

at8

public final long at8(long i)

Fetch element the slow way, as a long. Floating point values are silently rounded to an integer. Throws if the value is missing.

Returns:

ith element as a long, or throw if missing

at

public final double at(long i)

Fetch element the slow way, as a double, or Double.NaN is missing.

Returns:

ith element as a double, or Double.NaN if missing

isNA

public final boolean isNA(long row)

Fetch the missing-status the slow way.

Returns:

the missing-status the slow way

at16l

public final long at16l(long i)

Fetch element the slow way, as the low half of a UUID. Throws if the value is missing or not a UUID.

Returns:

ith element as a UUID low half, or throw if missing

at16h

public final long at16h(long i)

Fetch element the slow way, as the high half of a UUID. Throws if the value is missing or not a UUID.

Returns:

ith element as a UUID high half, or throw if missing

atStr

public final BufferedString atStr(BufferedString bStr,
                   long i)

Fetch element the slow way, as a BufferedString or null if missing. Throws if the value is not a String. BufferedStrings are String-like objects than can be reused in-place, which is much more efficient than constructing Strings.

Returns:

ith element as BufferedString or null if missing, or throw if not a String

set

public final void set(long i,
       long l)

Write element the slow way, as a long. There is no way to write a missing value with this call. Under rare circumstances this can throw: if the long does not fit in a double (value is larger magnitude than 2^52), AND float values are stored in Vec. In this case, there is no common compatible data representation.

set

public final void set(long i,
       double d)

Write element the slow way, as a double. Double.NaN will be treated as a set of a missing element.

set

public final void set(long i,
       float f)

Write element the slow way, as a float. Float.NaN will be treated as a set of a missing element.

set

public final void set(long i,
       java.lang.String str)

Write element the slow way, as a String. null will be treated as a set of a missing element.

open

public final Vec.Writer open()

Create a writer for bulk serial writes into this Vec.

Returns:

A Writer for bulk serial writes

toString

public java.lang.String toString()

Pretty print the Vec: [#elems, min/mean/max]{chunks,...}

Overrides:

toString in class java.lang.Object

Returns:

Brief string representation of a Vec

toCategoricalVec

public Vec toCategoricalVec()

Convenience method for converting to a categorical vector.

Returns:

A categorical vector based on the contents of the original vector.

toStringVec

public Vec toStringVec()

Convenience method for converting to a string vector.

Returns:

A string vector based on the contents of the original vector.

toNumericVec

public Vec toNumericVec()

Convenience method for converting to a numeric vector.

Returns:

A numeric vector based on the contents of the original vector.

equals

public boolean equals(java.lang.Object o)

True if two Vecs are equal. Checks for equal-Keys only (so it is fast) and not equal-contents.

Overrides:

equals in class java.lang.Object

Returns:

True if two Vecs are equal

hashCode

public int hashCode()

Vec's hashcode, which is just the Vec Key hashcode.

Overrides:

hashCode in class java.lang.Object

Returns:

Vec's hashcode

remove_impl

public Futures remove_impl(Futures fs)

Remove associated Keys when this guy removes. For Vecs, remove all associated Chunks.

Overrides:

remove_impl in class Keyed<Vec>

Returns:

Passed in Futures for flow-coding

writeAll_impl

protected AutoBuffer writeAll_impl(AutoBuffer ab)

Write out K/V pairs

Overrides:

writeAll_impl in class Keyed<Vec>

readAll_impl

protected Keyed readAll_impl(AutoBuffer ab,
                 Futures fs)

Overrides:

readAll_impl in class Keyed<Vec>

align

public Vec align(Vec vec)

Always makes a copy of the given vector which shares the same group as this Vec. This can be expensive operation since it can force copy of data among nodes.

Parameters:

vec - vector which is intended to be copied

Returns:

a copy of vec which shared the same Vec.VectorGroup with this vector

adaptTo

public CategoricalWrappedVec adaptTo(java.lang.String[] domain)

Make a Vec adapting this cal vector to the 'to' categorical Vec. The adapted CategoricalWrappedVec has 'this' as it's masterVec, but returns results in the 'to' domain (or just past it, if 'this' has elements not appearing in the 'to' domain).