Vec

java.lang.Object
- water.Iced
- - water.fvec.Vec

All Implemented Interfaces:: java.lang.Cloneable, Freezable

Direct Known Subclasses:: AppendableVec, ByteVec, WrappedVec

public class Vec
extends Iced

A single distributed vector column.

A distributed vector has a count of elements, an element-to-chunk mapping, a Java type (mostly determines rounding on store and display), and functions to directly load elements without further indirections. The data is compressed, or backed by disk or both. *Writing* to elements may throw if the backing data is read-only (file backed).

  Vec Key format is: Key. VEC - byte, 0 - byte,   0    - int, normal Key bytes.
 DVec Key format is: Key.DVEC - byte, 0 - byte, chunk# - int, normal Key bytes.

The main API is at, set, and isNA:

   double  at  ( long row );  // Returns the value expressed as a double.  NaN if missing.
   long    at8 ( long row );  // Returns the value expressed as a long.  Throws if missing.
   boolean isNA( long row );  // True if the value is missing.
   set( long row, double d ); // Stores a double; NaN will be treated as missing.
   set( long row, long l );   // Stores a long; throws if l exceeds what fits in a double and any floats are ever set.
   setNA( long row );         // Sets the value as missing.

Note this dangerous scenario: loading a missing value as a double, and setting it as a long:

   set(row,(long)at(row)); // Danger!

The cast from a Double.NaN to a long produces a zero! This code will replace a missing value with a zero.

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static class`	`Vec.CollectDomain` Collect numeric domain of given vector
`static class`	`Vec.VectorGroup` Class representing the group of vectors.
`static class`	`Vec.Writer` More efficient way to write randomly to a Vec - still slow, but much faster than Vec.set() Usage: Vec.Writer vw = vec.open(); vw.set(0, 3.32); vw.set(1, 4.32); vw.set(2, 5.32); vw.close();

Field Summary

Fields
Modifier and Type	Field and Description
`java.lang.String[]`	`_domain` Enum/factor/categorical names.
`long[]`	`_espc` Element-start per chunk.
`Key`	`_key` Key mapping a Value which holds this Vec.
`byte`	`_time` Time parse, index into Utils.TIME_PARSE, or -1 for not-a-time
`static int`	`KEY_PREFIX_LEN`
`static int`	`LOG_CHK` Log-2 of Chunk size.
`static int`	`MAX_ENUM_SIZE` Maximal size of enum domain

Constructor Summary

Constructors
Modifier	Constructor and Description
	`Vec(Key key, long[] espc)` Main default constructor; requires the caller understand Chunk layout already, along with count of missing elements.
	`Vec(Key key, long[] espc, java.lang.String[] domain)`
`protected`	`Vec(Key key, Vec v)`

Method Summary

Methods
Modifier and Type	Method and Description
`Vec`	`align(Vec vec)` Always makes a copy of the given vector which shares the same group.
`double`	`at(long i)` Fetch element the slow way, as a double.
`long`	`at8(long i)` Fetch element the slow way, as a long.
`long`	`byteSize()` Size of compressed vector data.
`int`	`cardinality()` Returns cardinality for enum domain or -1 for other types.
`protected boolean`	`checkMissing(int cidx, Value val)`
`long`	`chunk2StartElem(int cidx)` Convert a chunk-index into a starting row #.
`Chunk`	`chunkForChunkIdx(int cidx)` The Chunk for a chunk#.
`Chunk`	`chunkForRow(long i)` The Chunk for a row#.
`Value`	`chunkIdx(int cidx)` Get a Chunk's Value by index.
`Key`	`chunkKey(int cidx)` Get a Chunk Key from a chunk-index.
`int`	`chunkLen(int cidx)` Number of rows in chunk.
`java.lang.String[]`	`domain()` Return an array of domains.
`java.lang.String`	`domain(long i)` Map the integer value for a enum/factor/categorical to it's String.
`boolean`	`equals(java.lang.Object o)`
`static Key`	`getVecKey(Key key)` Get a Vec Key from Chunk Key, without loading the Chunk
`Vec.VectorGroup`	`group()` Get the group this vector belongs to.
`Key`	`groupKey()` Make a Vector-group key.
`byte[]`	`hash()`
`int`	`hashCode()`
`boolean`	`isBad()` Is the column bad.
`boolean`	`isByteVec()`
`boolean`	`isConst()` Is the column constant.
`boolean`	`isEnum()` Is the column a factor/categorical/enum? Note: all "isEnum()" columns are are also "isInt()" but not vice-versa.
`boolean`	`isFloat()` Is the column contains float values.
`boolean`	`isInt()` Is all integers?
`boolean`	`isNA(long row)` Fetch the missing-status the slow way.
`boolean`	`isTime()` Whether or not this column parsed as a time, and if so what pattern was used.
`long`	`length()` Number of elements in the vector.
`static Vec`	`make1Elem(double d)` Create a new 1-element vector in the shared vector group for 1-element vectors.
`static Vec`	`make1Elem(Key key, double d)` Create a new 1-element vector representing a scalar value.
`Vec`	`makeCon(double d)`
`Vec`	`makeCon(long l)` Make a new vector with the same size and data layout as the old one, and initialized to a constant.
`Vec`	`makeCon(long l, java.lang.String[] domain)`
`Vec[]`	`makeCons(int n, long l, java.lang.String[][] domain)`
`static Vec`	`makeConSeq(double x, int len)`
`static Vec[]`	`makeNewCons(long rows, int cols, long val, java.lang.String[][] domain)` Create an array of Vecs from scratch
`static Vec`	`makeSeq(int len)`
`Vec`	`makeTransf(int[][] map, java.lang.String[] finalDomain)` Create a vector transforming values according given domain map.
`Vec`	`makeZero()` Make a new vector with the same size and data layout as the old one, and initialized to zero.
`Vec`	`makeZero(java.lang.String[] domain)`
`Vec[]`	`makeZeros(int n)`
`Vec[]`	`makeZeros(int n, java.lang.String[][] domain)`
`Vec`	`masterVec()` This Vec does not have dependent hidden Vec it uses.
`double`	`max()` Return column max - lazily computed as needed.
`double`	`mean()` Return column mean - lazily computed as needed.
`double`	`min()` Return column min - lazily computed as needed.
`long`	`naCnt()` Return column missing-element-count - lazily computed as needed.
`int`	`nChunks()` Number of chunks.
`static Key`	`newKey()` Make a new random Key that fits the requirements for a Vec key.
`Vec.Writer`	`open()`
`void`	`postWrite()` Stop writing into this Vec.
`protected boolean`	`readable()` Default read/write behavior for Vecs.
`void`	`remove(Futures fs)`
`Vec`	`rollupStats()` Compute the roll-up stats as-needed, and copy into the Vec object
`Vec`	`rollupStats(Futures fs)`
`double`	`set(long i, double d)` Write element the VERY slow way, as a double.
`float`	`set(long i, float f)` Write element the VERY slow way, as a float.
`long`	`set(long i, long l)` Write element the VERY slow way, as a long.
`boolean`	`setNA(long i)` Set the element as missing the VERY slow way.
`double`	`sigma()` Return column standard deviation - lazily computed as needed.
`int`	`timeMode()`
`java.lang.String`	`timeParse()`
`Vec`	`toEnum()` Transform this vector to enum.
`java.lang.String`	`toString()` Pretty print the Vec: [#elems, min/mean/max]{chunks,...}
`protected boolean`	`writable()` Default read/write behavior for Vecs.

Methods inherited from class water.Iced
clone, frozenType, init, newInstance, read, toDocField, write, writeJSON, writeJSONFields

Methods inherited from class java.lang.Object
finalize, getClass, notify, notifyAll, wait, wait, wait

- Field Detail
  - LOG_CHK
```
public static final int LOG_CHK
```
    Log-2 of Chunk size.
    
    See Also:
    Constant Field Values
  - _key
```
public final Key _key
```
    Key mapping a Value which holds this Vec.
  - _espc
```
public final long[] _espc
```
    Element-start per chunk. Always zero for chunk 0. One more entry than chunks, so the last entry is the total number of rows. This field is dead/ignored in subclasses that are guaranteed to have fixed-sized chunks such as file-backed Vecs.
  - _domain
```
public java.lang.String[] _domain
```
    Enum/factor/categorical names.
  - _time
```
public byte _time
```
    Time parse, index into Utils.TIME_PARSE, or -1 for not-a-time
  - MAX_ENUM_SIZE
```
public static final int MAX_ENUM_SIZE
```
    Maximal size of enum domain
    
    See Also:
    Constant Field Values
  - KEY_PREFIX_LEN
```
public static final int KEY_PREFIX_LEN
```
    See Also:
    Constant Field Values
- Constructor Detail
  - Vec
```
public Vec(Key key,
   long[] espc)
```
    Main default constructor; requires the caller understand Chunk layout already, along with count of missing elements.
  - Vec
```
public Vec(Key key,
   long[] espc,
   java.lang.String[] domain)
```
  - Vec
```
protected Vec(Key key,
   Vec v)
```
- Method Detail
  - makeZeros
```
public Vec[] makeZeros(int n)
```
  - makeZeros
```
public Vec[] makeZeros(int n,
              java.lang.String[][] domain)
```
  - makeCons
```
public Vec[] makeCons(int n,
             long l,
             java.lang.String[][] domain)
```
  - makeNewCons
```
public static Vec[] makeNewCons(long rows,
                int cols,
                long val,
                java.lang.String[][] domain)
```
    Create an array of Vecs from scratch
    
    Parameters:
    rows - Length of each vec
    cols - Number of vecs
    val - Constant value (long)
    domain - Factor levels (for factor columns)
    
    Returns:
    Array of Vecs
  - makeZero
```
public Vec makeZero()
```
    Make a new vector with the same size and data layout as the old one, and initialized to zero.
  - makeZero
```
public Vec makeZero(java.lang.String[] domain)
```
  - makeCon
```
public Vec makeCon(long l)
```
    Make a new vector with the same size and data layout as the old one, and initialized to a constant.
  - makeCon
```
public Vec makeCon(long l,
          java.lang.String[] domain)
```
  - makeCon
```
public Vec makeCon(double d)
```
  - makeSeq
```
public static Vec makeSeq(int len)
```
  - makeConSeq
```
public static Vec makeConSeq(double x,
             int len)
```
  - make1Elem
```
public static Vec make1Elem(double d)
```
    Create a new 1-element vector in the shared vector group for 1-element vectors.
  - make1Elem
```
public static Vec make1Elem(Key key,
            double d)
```
    Create a new 1-element vector representing a scalar value.
  - makeTransf
```
public Vec makeTransf(int[][] map,
             java.lang.String[] finalDomain)
```
    Create a vector transforming values according given domain map.
    
    See Also:
    makeTransf(int[], int[], String[])
  - masterVec
```
public Vec masterVec()
```
    This Vec does not have dependent hidden Vec it uses.
    
    Returns:
    dependent hidden vector or null
  - length
```
public long length()
```
    Number of elements in the vector. Overridden by subclasses that compute length in an alternative way, such as file-backed Vecs.
  - nChunks
```
public int nChunks()
```
    Number of chunks. Overridden by subclasses that compute chunks in an alternative way, such as file-backed Vecs.
  - isTime
```
public final boolean isTime()
```
    Whether or not this column parsed as a time, and if so what pattern was used.
  - timeMode
```
public final int timeMode()
```
  - timeParse
```
public final java.lang.String timeParse()
```
  - domain
```
public java.lang.String domain(long i)
```
    Map the integer value for a enum/factor/categorical to it's String. Error if it is not an ENUM.
  - domain
```
public java.lang.String[] domain()
```
    Return an array of domains. This is eagerly manifested for enum or categorical columns. Returns null for non-Enum/factor columns.
  - cardinality
```
public int cardinality()
```
    Returns cardinality for enum domain or -1 for other types.
  - toEnum
```
public Vec toEnum()
```
    Transform this vector to enum. If the vector is integer vector then its domain is collected and transformed to corresponding strings. If the vector is enum an identity transformation vector is returned. Transformation is done by a TransfVec which provides a mapping between values.
    
    Returns:
    always returns a new vector and the caller is responsible for vector deletion!
  - readable
```
protected boolean readable()
```
    Default read/write behavior for Vecs. File-backed Vecs are read-only.
  - writable
```
protected boolean writable()
```
    Default read/write behavior for Vecs. AppendableVecs are write-only.
  - min
```
public double min()
```
    Return column min - lazily computed as needed.
  - max
```
public double max()
```
    Return column max - lazily computed as needed.
  - mean
```
public double mean()
```
    Return column mean - lazily computed as needed.
  - sigma
```
public double sigma()
```
    Return column standard deviation - lazily computed as needed.
  - naCnt
```
public long naCnt()
```
    Return column missing-element-count - lazily computed as needed.
  - isInt
```
public boolean isInt()
```
    Is all integers?
  - byteSize
```
public long byteSize()
```
    Size of compressed vector data.
  - hash
```
public byte[] hash()
```
  - isEnum
```
public final boolean isEnum()
```
    Is the column a factor/categorical/enum? Note: all "isEnum()" columns are are also "isInt()" but not vice-versa.
  - isConst
```
public final boolean isConst()
```
    Is the column constant.
    Returns true if the column contains only constant values and it is not full of NAs.
  - isBad
```
public final boolean isBad()
```
    Is the column bad.
    Returns true if the column is full of NAs.
  - isFloat
```
public final boolean isFloat()
```
    Is the column contains float values.
  - isByteVec
```
public final boolean isByteVec()
```
  - rollupStats
```
public Vec rollupStats()
```
    Compute the roll-up stats as-needed, and copy into the Vec object
  - rollupStats
```
public Vec rollupStats(Futures fs)
```
  - postWrite
```
public void postWrite()
```
    Stop writing into this Vec. Rollup stats will again (lazily) be computed.
  - chunk2StartElem
```
public long chunk2StartElem(int cidx)
```
    Convert a chunk-index into a starting row #. For constant-sized chunks this is a little shift-and-add math. For variable-sized chunks this is a table lookup.
  - chunkLen
```
public int chunkLen(int cidx)
```
    Number of rows in chunk. Does not fetch chunk content.
  - getVecKey
```
public static Key getVecKey(Key key)
```
    Get a Vec Key from Chunk Key, without loading the Chunk
  - chunkKey
```
public Key chunkKey(int cidx)
```
    Get a Chunk Key from a chunk-index. Basically the index-to-key map.
  - chunkIdx
```
public Value chunkIdx(int cidx)
```
    Get a Chunk's Value by index. Basically the index-to-key map, plus the DKV.get(). Warning: this pulls the data locally; using this call on every Chunk index on the same node will probably trigger an OOM!
  - checkMissing
```
protected boolean checkMissing(int cidx,
                   Value val)
```
  - newKey
```
public static Key newKey()
```
    Make a new random Key that fits the requirements for a Vec key.
  - groupKey
```
public Key groupKey()
```
    Make a Vector-group key.
  - group
```
public final Vec.VectorGroup group()
```
    Get the group this vector belongs to. In case of a group with only one vector, the object actually does not exist in KV store.
    
    Returns:
    VectorGroup this vector belongs to.
  - chunkForChunkIdx
```
public Chunk chunkForChunkIdx(int cidx)
```
    The Chunk for a chunk#. Warning: this loads the data locally!
  - chunkForRow
```
public final Chunk chunkForRow(long i)
```
    The Chunk for a row#. Warning: this loads the data locally!
  - at8
```
public final long at8(long i)
```
    Fetch element the slow way, as a long. Floating point values are silently rounded to an integer. Throws if the value is missing.
  - at
```
public final double at(long i)
```
    Fetch element the slow way, as a double. Missing values are returned as Double.NaN instead of throwing.
  - isNA
```
public final boolean isNA(long row)
```
    Fetch the missing-status the slow way.
  - set
```
public final long set(long i,
       long l)
```
    Write element the VERY slow way, as a long. There is no way to write a missing value with this call. Under rare circumstances this can throw: if the long does not fit in a double (value is larger magnitude than 2^52), AND float values are stored in Vector. In this case, there is no common compatible data representation. NOTE: For a faster way, but still slow, use the Vec.Writer below.
  - set
```
public final double set(long i,
         double d)
```
    Write element the VERY slow way, as a double. Double.NaN will be treated as a set of a missing element.
  - set
```
public final float set(long i,
        float f)
```
    Write element the VERY slow way, as a float. Float.NaN will be treated as a set of a missing element.
  - setNA
```
public final boolean setNA(long i)
```
    Set the element as missing the VERY slow way.
  - open
```
public final Vec.Writer open()
```
  - toString
```
public java.lang.String toString()
```
    Pretty print the Vec: [#elems, min/mean/max]{chunks,...}
    
    Overrides:
    
    toString in class java.lang.Object
  - remove
```
public void remove(Futures fs)
```
  - equals
```
public boolean equals(java.lang.Object o)
```
    Overrides:
    
    equals in class java.lang.Object
  - hashCode
```
public int hashCode()
```
    Overrides:
    
    hashCode in class java.lang.Object
  - align
```
public Vec align(Vec vec)
```
    Always makes a copy of the given vector which shares the same group. The user is responsible for deleting the returned vector. This can be expensive operation since it can force copy of data among nodes.
    
    Parameters:
    vec - vector which is intended to be copied
    
    Returns:
    a copy of vec which shared the same Vec.VectorGroup with this vector

Class Vec

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class water.Iced

Methods inherited from class java.lang.Object

Field Detail

LOG_CHK

_key

_espc

_domain

_time

MAX_ENUM_SIZE

KEY_PREFIX_LEN

Constructor Detail

Vec

Vec

Vec

Method Detail

makeZeros

makeZeros

makeCons

makeNewCons

makeZero

makeZero

makeCon

makeCon

makeCon

makeSeq

makeConSeq

make1Elem

make1Elem

makeTransf

masterVec

length

nChunks

isTime

timeMode

timeParse

domain

domain

cardinality

toEnum

readable

writable

min

max

mean

sigma

naCnt

isInt

byteSize

hash

isEnum

isConst

isBad

isFloat

isByteVec

rollupStats

rollupStats

postWrite

chunk2StartElem

chunkLen

getVecKey

chunkKey

chunkIdx

checkMissing

newKey

groupKey

group

chunkForChunkIdx

chunkForRow

at8

at

isNA

set

set

set

setNA