hex.deeplearning

Class Neurons

java.lang.Object

hex.deeplearning.Neurons

Direct Known Subclasses:

Neurons.Input, Neurons.Maxout, Neurons.Output, Neurons.Rectifier, Neurons.Tanh

public abstract class Neurons
extends java.lang.Object

This class implements the concept of a Neuron layer in a Neural Network During training, every MRTask F/J thread is expected to create these neurons for every map call (Cheap to make). These Neurons are NOT sent over the wire. The weights connecting the neurons are in a separate class (DeepLearningModel.DeepLearningModelInfo), and will be shared per node.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Neurons.DenseColMatrix Dense column matrix implementation
static class	Neurons.DenseRowMatrix Dense row matrix implementation
static class	Neurons.DenseVector Dense vector implementation
static class	Neurons.Input Input layer of the Neural Network This layer is different from other layers as it has no incoming weights, but instead gets its activation values from the training points.
static class	Neurons.Linear Output neurons for regression - Linear units
static interface	Neurons.Matrix Abstract matrix interface
static class	Neurons.Maxout Maxout neurons
static class	Neurons.MaxoutDropout Maxout neurons with dropout
static class	Neurons.Output Abstract class for Output neurons
static class	Neurons.Rectifier Rectifier linear unit (ReLU) neurons
static class	Neurons.RectifierDropout Rectifier linear unit (ReLU) neurons with dropout
static class	Neurons.Softmax Output neurons for classification - Softmax
static class	Neurons.SparseRowMatrix Sparse row matrix implementation
static class	Neurons.SparseVector Sparse vector implementation
static class	Neurons.Tanh Tanh neurons - most common, most stable
static class	Neurons.TanhDropout Tanh neurons with dropout
static interface	Neurons.Vector Abstract vector interface

Field Summary

Fields

Modifier and Type	Field and Description
Neurons.Vector	_a Layer state (one per neuron): activity, error
Neurons.DenseVector	_avg_a
Neurons.DenseVector	_b
protected Dropout	_dropout For Dropout training
Neurons.DenseVector	_e
protected int	_index
Neurons	_input
Neurons	_previous References for feed-forward connectivity
Neurons.Matrix	_w
static int	missing_int_value
static java.lang.Float	missing_real_value
protected DeepLearningModel.DeepLearningParameters	params Parameters (deep-cloned() from the user input, can be modified here, e.g.
protected int	units

Method Summary

Methods

Modifier and Type	Method and Description
protected float	autoEncoderError(int row) Helper to compute the reconstruction error for auto-encoders (part of the gradient computation)
protected abstract void	bprop() Back propagation
protected abstract void	fprop(long seed, boolean training) Forward propagation
void	init(Neurons[] neurons, int index, DeepLearningModel.DeepLearningParameters p, DeepLearningModel.DeepLearningModelInfo minfo, boolean training) Initialization of the parameters and connectivity of a Neuron layer
protected float	momentum()
float	momentum(long n) The momentum - real number in [0, 1) Can be a linear ramp from momentum_start to momentum_stable, over momentum_ramp training samples
float	rate(long n) The learning rate
java.lang.String	toString() Print the status of this neuron layer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

units

protected int units

params

protected transient DeepLearningModel.DeepLearningParameters params

Parameters (deep-cloned() from the user input, can be modified here, e.g. learning rate decay)

_index

protected transient int _index

_a

public transient Neurons.Vector _a

Layer state (one per neuron): activity, error

_e

public transient Neurons.DenseVector _e

_previous

public Neurons _previous

References for feed-forward connectivity

_input

public Neurons _input

_w

public Neurons.Matrix _w

_b

public Neurons.DenseVector _b

_dropout

protected Dropout _dropout

For Dropout training

_avg_a

public Neurons.DenseVector _avg_a

missing_int_value

public static final int missing_int_value

See Also:

Constant Field Values

missing_real_value

public static final java.lang.Float missing_real_value

Method Detail

toString

public java.lang.String toString()

Print the status of this neuron layer

Overrides:

toString in class java.lang.Object

Returns:

populated String

init

public final void init(Neurons[] neurons,
        int index,
        DeepLearningModel.DeepLearningParameters p,
        DeepLearningModel.DeepLearningModelInfo minfo,
        boolean training)

Initialization of the parameters and connectivity of a Neuron layer

Parameters:

neurons - Array of all neuron layers, to establish feed-forward connectivity

index - Which layer am I?

p - User-given parameters (Job parental object hierarchy is not used)

minfo - Model information (weights/biases and their momenta)

training - Whether training is done or just testing (no need for dropout)

fprop

protected abstract void fprop(long seed,
         boolean training)

Forward propagation

Parameters:

seed - For seeding the RNG inside (for dropout)

training - Whether training is done or just testing (no need for dropout)

bprop

protected abstract void bprop()

Back propagation

autoEncoderError

protected float autoEncoderError(int row)

Helper to compute the reconstruction error for auto-encoders (part of the gradient computation)

Parameters:

row - neuron index

Returns:

difference between the output (auto-encoder output layer activation) and the target (input layer activation)

rate

public float rate(long n)

The learning rate

Parameters:

n - The number of training samples seen so far (for rate_annealing greater than 0)

Returns:

Learning rate

momentum

protected float momentum()

momentum

public float momentum(long n)

The momentum - real number in [0, 1) Can be a linear ramp from momentum_start to momentum_stable, over momentum_ramp training samples

Parameters:

n - The number of training samples seen so far

Returns:

momentum