water.parser

Class ParseSetup.GuessSetupTsk

java.lang.Object

jsr166y.ForkJoinTask<java.lang.Void>

jsr166y.CountedCompleter

water.H2O.H2OCountedCompleter<T>

water.DTask<T>

water.MRTask<ParseSetup.GuessSetupTsk>

water.parser.ParseSetup.GuessSetupTsk

All Implemented Interfaces:

java.io.Serializable, java.lang.Cloneable, java.util.concurrent.Future<java.lang.Void>, ForkJoinPool.ManagedBlocker, Freezable<ParseSetup.GuessSetupTsk>

Enclosing class:

ParseSetup

public static class ParseSetup.GuessSetupTsk
extends MRTask<ParseSetup.GuessSetupTsk>

Try to determine the ParseSetup on a file by file basis and merge results.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class water.DTask

DTask.DKeyTask<T extends DTask.DKeyTask,V extends Keyed>, DTask.RemoveCall

Field Summary

Fields

Modifier and Type	Field and Description
ParseSetup	_gblSetup
long	_maxLineLength
long	_totalParseSize

Fields inherited from class water.MRTask

_appendables, _fr, _fs, _hi, _keys, _left, _lo, _nhi, _nleft, _nlo, _nrite, _rite, _run_local, _topLocal

Fields inherited from class water.DTask

_ex

Constructor Summary

Constructors

Constructor and Description
ParseSetup.GuessSetupTsk(ParseSetup userSetup)

Method Summary

Methods

Modifier and Type	Method and Description
void	map(Key key) Runs once on each file to guess that file's ParseSetup For ByteVecs, UploadFileVecs, compressed files and small files, the ParseSetup is guessed from a single DFLT_CHUNK_SIZE chunk from the start of the file.
void	postGlobal()
void	reduce(ParseSetup.GuessSetupTsk other) Merges ParseSetup results, conflicts, and errors from several files

Methods inherited from class water.MRTask

appendables, asyncExecOnAllNodes, block, closeLocal, compute2, dfork, dfork, dfork, dfork, dfork, dinvoke, doAll, doAll, doAll, doAll, doAll, doAll, doAll, doAll, doAll, doAll, doAll, doAll, doAllNodes, getResult, getResult, isReleasable, map, map, map, map, map, map, map, map, map, map, modifiesVolatileVecs, onCompletion, onExceptionalCompletion, outputFrame, outputFrame, outputFrame, profile, profString, self, setupLocal

Methods inherited from class water.DTask

copyOver, getDException, hasException, logVerbose, onAck, onAckAck, setException

Methods inherited from class water.H2O.H2OCountedCompleter

asBytes, clone, compute, compute1, currThrPriority, frozenType, icer, priority, read, readJSON, reloadFromBytes, write, writeJSON

Methods inherited from class jsr166y.CountedCompleter

__tryComplete, addToPendingCount, compareAndSetPendingCount, complete, exec, getCompleter, getPendingCount, getRawResult, setCompleter, setPendingCount, setRawResult, tryComplete

Methods inherited from class jsr166y.ForkJoinTask

adapt, adapt, adapt, cancel, compareAndSetForkJoinTaskTag, completeExceptionally, fork, get, get, getException, getForkJoinTaskTag, getPool, getQueuedTaskCount, getSurplusQueuedTaskCount, helpQuiesce, inForkJoinPool, invoke, invokeAll, invokeAll, invokeAll, isCancelled, isCompletedAbnormally, isCompletedNormally, isDone, join, peekNextLocalTask, pollNextLocalTask, pollTask, quietlyComplete, quietlyInvoke, quietlyJoin, reinitialize, setForkJoinTaskTag, tryUnfork

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

_gblSetup

public ParseSetup _gblSetup

_totalParseSize

public long _totalParseSize

_maxLineLength

public long _maxLineLength

Constructor Detail

ParseSetup.GuessSetupTsk

public ParseSetup.GuessSetupTsk(ParseSetup userSetup)

Parameters:

userSetup - ParseSetup to guide examination of files

Method Detail

map

public void map(Key key)

Runs once on each file to guess that file's ParseSetup For ByteVecs, UploadFileVecs, compressed files and small files, the ParseSetup is guessed from a single DFLT_CHUNK_SIZE chunk from the start of the file. This is because UploadFileVecs and compressed files don't allow random sampling, small files don't need it, and ByteVecs tend to be small. For larger NSFFileVecs and HDFSFileVecs 1M samples are taken at the beginning of every 100M, and an additional sample is taken from the last chunk of the file. The results of these samples are merged together (and compared for consistency). Sampling more than the first bytes is preferred, since large data sets with sorted columns may have all the same value in their first bytes, making for poor type guesses.

Overrides:

map in class MRTask<ParseSetup.GuessSetupTsk>

reduce

public void reduce(ParseSetup.GuessSetupTsk other)

Merges ParseSetup results, conflicts, and errors from several files

Overrides:

reduce in class MRTask<ParseSetup.GuessSetupTsk>

postGlobal

public void postGlobal()

Overrides:

postGlobal in class MRTask<ParseSetup.GuessSetupTsk>