hex.deepwater

Class DeepWaterTask

java.lang.Object

jsr166y.ForkJoinTask<java.lang.Void>

jsr166y.CountedCompleter

water.H2O.H2OCountedCompleter<T>

water.DTask<T>

water.MRTask<T>

hex.FrameTask<DeepWaterTask>

hex.deepwater.DeepWaterTask

All Implemented Interfaces:

java.io.Serializable, java.lang.Cloneable, java.util.concurrent.Future<java.lang.Void>, jsr166y.ForkJoinPool.ManagedBlocker, water.Freezable<DeepWaterTask>

public class DeepWaterTask
extends FrameTask<DeepWaterTask>

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class hex.FrameTask

FrameTask.ExtractDenseRow

Nested classes/interfaces inherited from class water.DTask

water.DTask.DKeyTask<T extends water.DTask.DKeyTask,V extends water.Keyed>, water.DTask.RemoveCall

Field Summary

Fields inherited from class hex.FrameTask

_dinfo, _jobKey, _sparse

Fields inherited from class water.MRTask

_appendables, _fr, _fs, _hi, _keys, _left, _lo, _nhi, _nleft, _nlo, _nrite, _rite, _run_local, _topLocal

Fields inherited from class water.DTask

_ex

Method Summary

Methods

Modifier and Type	Method and Description
protected void	closeLocal() After all maps are done on a node, this is called to store the per-node model into DKV (for elastic averaging) Otherwise, do nothing.
void	map(water.fvec.Chunk[] chunks, water.fvec.NewChunk[] outputs) Extracts the values, applies regularization to numerics, adds appropriate offsets to categoricals, and adapts response according to the CaseMode/CaseValue if set.
DeepWaterModelInfo	model_info() Accessor to the object containing the (final) state of the Deep Learning model Should only be queried after calling this.doAll(Frame training)
protected void	postGlobal() After all reduces are done, the driver node calls this method to clean up This is only needed if we're not inside a DeepWaterTask2 (which will do the reduction between replicated data workers).
void	reduce(DeepWaterTask other) Average the per-node models (for elastic averaging, already wrote them to DKV in postLocal()) This is a no-op between F/J worker threads (operate on the same weights/biases)
protected void	setupLocal() Transfer ownership from global (shared) model to local model which will be worked on

Methods inherited from class hex.FrameTask

chunkDone, chunkInit, dinfo, getMiniBatchSize, processMiniBatch, processRow, processRow, processRow

Methods inherited from class water.MRTask

appendables, asyncExecOnAllNodes, block, compute2, dfork, dfork, dfork, dfork, dfork, dinvoke, doAll, doAll, doAll, doAll, doAll, doAll, doAll, doAll, doAll, doAll, doAll, doAll, doAllNodes, getResult, getResult, isReleasable, map, map, map, map, map, map, map, map, map, map, modifiesVolatileVecs, onCompletion, onExceptionalCompletion, outputFrame, outputFrame, outputFrame, profile, profString, self

Methods inherited from class water.DTask

copyOver, getDException, hasException, logVerbose, onAck, onAckAck, setException

Methods inherited from class water.H2O.H2OCountedCompleter

asBytes, clone, compute, compute1, currThrPriority, frozenType, icer, priority, read, readJSON, reloadFromBytes, write, writeJSON

Methods inherited from class jsr166y.CountedCompleter

__tryComplete, addToPendingCount, compareAndSetPendingCount, complete, exec, getCompleter, getPendingCount, getRawResult, setCompleter, setPendingCount, setRawResult, tryComplete

Methods inherited from class jsr166y.ForkJoinTask

adapt, adapt, adapt, cancel, compareAndSetForkJoinTaskTag, completeExceptionally, fork, get, get, getException, getForkJoinTaskTag, getPool, getQueuedTaskCount, getSurplusQueuedTaskCount, helpQuiesce, inForkJoinPool, invoke, invokeAll, invokeAll, invokeAll, isCancelled, isCompletedAbnormally, isCompletedNormally, isDone, join, peekNextLocalTask, pollNextLocalTask, pollTask, quietlyComplete, quietlyInvoke, quietlyJoin, reinitialize, setForkJoinTaskTag, tryUnfork

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

model_info

public final DeepWaterModelInfo model_info()

Accessor to the object containing the (final) state of the Deep Learning model Should only be queried after calling this.doAll(Frame training)

Returns:

"The" final model after one Map/Reduce iteration

setupLocal

protected void setupLocal()

Transfer ownership from global (shared) model to local model which will be worked on

Overrides:

setupLocal in class FrameTask<DeepWaterTask>

map

public void map(water.fvec.Chunk[] chunks,
       water.fvec.NewChunk[] outputs)

Description copied from class: FrameTask

Extracts the values, applies regularization to numerics, adds appropriate offsets to categoricals, and adapts response according to the CaseMode/CaseValue if set.

Overrides:

map in class FrameTask<DeepWaterTask>

closeLocal

protected void closeLocal()

After all maps are done on a node, this is called to store the per-node model into DKV (for elastic averaging) Otherwise, do nothing.

Overrides:

closeLocal in class FrameTask<DeepWaterTask>

reduce

public void reduce(DeepWaterTask other)

Average the per-node models (for elastic averaging, already wrote them to DKV in postLocal()) This is a no-op between F/J worker threads (operate on the same weights/biases)

Overrides:

reduce in class water.MRTask<DeepWaterTask>

Parameters:

other - Other DeepWaterTask to reduce

postGlobal

protected void postGlobal()

After all reduces are done, the driver node calls this method to clean up This is only needed if we're not inside a DeepWaterTask2 (which will do the reduction between replicated data workers). So if replication is disabled, and every node works on partial data, then we have work to do here (model averaging).

Overrides:

postGlobal in class water.MRTask<DeepWaterTask>