water.util

Class MRUtils

java.lang.Object

water.util.MRUtils

public class MRUtils
extends java.lang.Object

Nested Class Summary

Nested Classes

Modifier and Type

Class and Description

static class

MRUtils.ClassDist Compute the class distribution from a class label vector (not counting missing values) Usage 1: Label vector is categorical ------------------------------------ Vec label = ...; assert(label.isEnum()); long[] dist = new ClassDist(label).doAll(label).dist(); Usage 2: Label vector is numerical ---------------------------------- Vec label = ...; int num_classes = ...; assert(label.isInt()); long[] dist = new ClassDist(num_classes).doAll(label).dist();

Constructor Summary

Constructors

Constructor and Description
MRUtils()

Method Summary

Methods

Modifier and Type	Method and Description
static Frame	sampleFrame(Frame fr, long rows, long seed) Sample rows from a frame.
static Frame	sampleFrameStratified(Frame fr, Vec label, float[] sampling_ratios, long seed, boolean debug) Stratified sampling
static Frame	sampleFrameStratified(Frame fr, Vec label, float[] sampling_ratios, long maxrows, long seed, boolean allowOversampling, boolean verbose) Stratified sampling for classifiers
static Frame	shuffleAndBalance(Frame fr, int splits, long seed, boolean local, boolean shuffle) Global redistribution of a Frame (balancing of chunks), done by calling process (all-to-one + one-to-all)
static Frame	shuffleFramePerChunk(Frame fr, long seed) Row-wise shuffle of a frame (only shuffles rows inside of each chunk)
static Frame	shuffleFramePerChunk(Key outputFrameKey, Frame fr, long seed)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MRUtils

public MRUtils()

Method Detail

sampleFrame

public static Frame sampleFrame(Frame fr,
                long rows,
                long seed)

Sample rows from a frame. Can be unlucky for small sampling fractions - will continue calling itself until at least 1 row is returned.

Parameters:

fr - Input frame

rows - Approximate number of rows to sample (across all chunks)

seed - Seed for RNG

Returns:

Sampled frame

shuffleFramePerChunk

public static Frame shuffleFramePerChunk(Frame fr,
                         long seed)

Row-wise shuffle of a frame (only shuffles rows inside of each chunk)

Parameters:

fr - Input frame

Returns:

Shuffled frame

shuffleFramePerChunk

public static Frame shuffleFramePerChunk(Key outputFrameKey,
                         Frame fr,
                         long seed)

shuffleAndBalance

public static Frame shuffleAndBalance(Frame fr,
                      int splits,
                      long seed,
                      boolean local,
                      boolean shuffle)

Global redistribution of a Frame (balancing of chunks), done by calling process (all-to-one + one-to-all)

Parameters:

fr - Input frame

seed - RNG seed

shuffle - whether to shuffle the data globally

Returns:

Shuffled frame

sampleFrameStratified

public static Frame sampleFrameStratified(Frame fr,
                          Vec label,
                          float[] sampling_ratios,
                          long maxrows,
                          long seed,
                          boolean allowOversampling,
                          boolean verbose)

Stratified sampling for classifiers

Parameters:

fr - Input frame

label - Label vector (must be enum)

sampling_ratios - Optional: array containing the requested sampling ratios per class (in order of domains), will be overwritten if it contains all 0s

maxrows - Maximum number of rows in the returned frame

seed - RNG seed for sampling

allowOversampling - Allow oversampling of minority classes

verbose - Whether to print verbose info

Returns:

Sampled frame, with approximately the same number of samples from each class (or given by the requested sampling ratios)

sampleFrameStratified

public static Frame sampleFrameStratified(Frame fr,
                          Vec label,
                          float[] sampling_ratios,
                          long seed,
                          boolean debug)

Stratified sampling

Parameters:

fr - Input frame

label - Label vector (from the input frame)

sampling_ratios - Given sampling ratios for each class, in order of domains

seed - RNG seed

debug - Whether to print debug info

Returns:

Stratified frame