water.fvec

Class AppendableVec

java.lang.Object

water.Iced<T>

water.Keyed<Vec>

water.fvec.Vec

water.fvec.AppendableVec

All Implemented Interfaces:

java.io.Externalizable, java.io.Serializable, java.lang.Cloneable, Freezable<Vec>

public class AppendableVec
extends Vec

A NEW single distributed vector column. The NEW vector has no data, and takes no space. It supports distributed parallel writes to it, via calls to append2. Such writes happen in parallel and all writes are ordered. Writes *will* be local to the node doing them, specifically to allow control over locality. By default, writes will go local-homed chunks with no compression; there is a final 'close' to the NEW vector which may do compression; the final 'close' will return some other Vec type. NEW Vectors do NOT support reads!

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class water.fvec.Vec

Vec.CollectDomain, Vec.CollectDomainFast, Vec.VectorGroup, Vec.Writer

Nested classes/interfaces inherited from class water.Keyed

Keyed.BinarySerializer<X extends Keyed>

Field Summary

Fields

Modifier and Type	Field and Description
int	_chunkOff
long[]	_espc
static byte	ENUM
static byte	NA
static byte	NUMBER
static byte	STRING
static byte	TIME
static byte	UUID

Fields inherited from class water.fvec.Vec

DO_HISTOGRAMS, KEY_PREFIX_LEN, NO_HISTOGRAMS, PERCENTILES, T_BAD, T_ENUM, T_NUM, T_STR, T_TIME, T_TIMELAST, T_UUID, TYPE_STR

Fields inherited from class water.Keyed

_key, EMPTY_KEY_LIST

Constructor Summary

Constructors

Constructor and Description
AppendableVec(Key key)
AppendableVec(Key key, long[] espc, int chunkOff)

Method Summary

Methods

Modifier and Type	Method and Description
long	byteSize() Size of compressed vector data.
protected long	chunk2StartElem(int cidx) Convert a chunk-index into a starting row #.
NewChunk	chunkForChunkIdx(int cidx) The Chunk for a chunk#.
Value	chunkIdx(int cidx) Get a Chunk's Value by index.
Vec	close(Futures fs)
static Vec[]	closeAll(AppendableVec[] avs)
static Vec[]	closeAll(AppendableVec[] avs, Futures fs)
long	length() Number of elements in the vector; returned as a long instead of an int because Vecs support more than 2^32 elements.
int	nChunks() Number of chunks, returned as an int - Chunk count is limited by the max size of a Java long[].
protected boolean	readable() Default read/write behavior for Vecs.
void	reduce(AppendableVec nv)
void	setPrecedingChunkTypes(int cidx, byte type) This is a hack to fix SVMLight parsing.
void	setSubRange(AppendableVec av) Add AV build over sub-range of this vec (used e.g.
void	setTypes(byte[] ts)
boolean	shouldBeEnum()
java.lang.String	toString() Pretty print the Vec: [#elems, min/mean/max]{chunks,...}
protected boolean	writable() Default read/write behavior for Vecs.

Methods inherited from class water.fvec.Vec

adaptTo, align, at, at16h, at16l, at8, atStr, base, bins, cardinality, checksum_impl, chunkForRow, chunkKey, chunkKey, domain, equals, factor, get_espc, get_type_str, get_type, getVecKey, group, hashCode, isBad, isBinary, isConst, isEnum, isInt, isNA, isNumeric, isString, isTime, isUUID, lazy_bins, makeCon, makeCon, makeCon, makeCon, makeCon, makeCon, makeCons, makeCopy, makeCopy, makeRand, makeRepSeq, makeSeq, makeSeq, makeVec, makeVec, makeZero, makeZero, makeZero, makeZero, makeZeros, makeZeros, max, maxs, mean, min, mins, naCnt, newKey, ninfs, nzCnt, open, pctiles, pinfs, postWrite, preWriting, remove_impl, set, set, set, set, setDomain, sigma, sparseRatio, startRollupStats, startRollupStats, stride, toByteArray, toDoubleArray, toEnum, toInt, toStringVec

Methods inherited from class water.Keyed

checksum, getBinarySerializer, getPublishedKeys, remove, remove, remove, remove

Methods inherited from class water.Iced

clone, frozenType, read_impl, read, readExternal, readJSON_impl, readJSON, toJsonString, write_impl, write, writeExternal, writeHTML_impl, writeHTML, writeJSON_impl, writeJSON

Methods inherited from class java.lang.Object

finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

_espc

public long[] _espc

NA

public static final byte NA

See Also:

Constant Field Values

ENUM

public static final byte ENUM

See Also:

Constant Field Values

NUMBER

public static final byte NUMBER

See Also:

Constant Field Values

TIME

public static final byte TIME

See Also:

Constant Field Values

UUID

public static final byte UUID

See Also:

Constant Field Values

STRING

public static final byte STRING

See Also:

Constant Field Values

_chunkOff

public int _chunkOff

Constructor Detail

AppendableVec

public AppendableVec(Key key)

AppendableVec

public AppendableVec(Key key,
             long[] espc,
             int chunkOff)

Method Detail

setTypes

public void setTypes(byte[] ts)

setPrecedingChunkTypes

public void setPrecedingChunkTypes(int cidx,
                          byte type)

This is a hack to fix SVMLight parsing. AppendableVec.close() checks the types of each chunk. This sets the chunk type for uncreated chunks of all 0, are counted as numeric and not NA.

Parameters:

cidx -

closeAll

public static Vec[] closeAll(AppendableVec[] avs)

closeAll

public static Vec[] closeAll(AppendableVec[] avs,
             Futures fs)

setSubRange

public void setSubRange(AppendableVec av)

Add AV build over sub-range of this vec (used e.g. by multifile parse where each file produces its own AV which represents sub-range of the resulting vec)

Parameters:

av -

shouldBeEnum

public boolean shouldBeEnum()

reduce

public void reduce(AppendableVec nv)

close

public Vec close(Futures fs)

readable

protected boolean readable()

Description copied from class: Vec

Default read/write behavior for Vecs. File-backed Vecs are read-only.

writable

protected boolean writable()

Description copied from class: Vec

Default read/write behavior for Vecs. AppendableVecs are write-only.

chunkForChunkIdx

public NewChunk chunkForChunkIdx(int cidx)

Description copied from class: Vec

The Chunk for a chunk#. Warning: this pulls the data locally; using this call on every Chunk index on the same node will probably trigger an OOM!

Overrides:

chunkForChunkIdx in class Vec

Returns:

Chunk for a chunk#

chunkIdx

public Value chunkIdx(int cidx)

Description copied from class: Vec

Get a Chunk's Value by index. Basically the index-to-key map, plus the DKV.get(). Warning: this pulls the data locally; using this call on every Chunk index on the same node will probably trigger an OOM!

Overrides:

chunkIdx in class Vec

length

public long length()

Description copied from class: Vec

Number of elements in the vector; returned as a long instead of an int because Vecs support more than 2^32 elements. Overridden by subclasses that compute length in an alternative way, such as file-backed Vecs.

Overrides:

length in class Vec

Returns:

Number of elements in the vector

nChunks

public int nChunks()

Description copied from class: Vec

Number of chunks, returned as an int - Chunk count is limited by the max size of a Java long[]. Overridden by subclasses that compute chunks in an alternative way, such as file-backed Vecs.

Overrides:

nChunks in class Vec

Returns:

Number of chunks

chunk2StartElem

protected long chunk2StartElem(int cidx)

Description copied from class: Vec

Convert a chunk-index into a starting row #. For constant-sized chunks this is a little shift-and-add math. For variable-sized chunks this is a table lookup.

byteSize

public long byteSize()

Description copied from class: Vec

Size of compressed vector data.

Overrides:

byteSize in class Vec

toString

public java.lang.String toString()

Description copied from class: Vec

Pretty print the Vec: [#elems, min/mean/max]{chunks,...}

Overrides:

toString in class Vec

Returns:

Brief string representation of a Vec