water.parser

Class ParseSetup

java.lang.Object

water.Iced

water.parser.ParseSetup

All Implemented Interfaces:

java.io.Externalizable, java.io.Serializable, java.lang.Cloneable, Freezable

public class ParseSetup
extends Iced

A generic configuration and base guesser for a parser.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ParseSetup.GuessSetupTsk Try to determine the ParseSetup on a file by file basis and merge results.

Field Summary

Fields

Modifier and Type	Field and Description
int	_chunk_size
ParseWriter.ParseErr[]	_errs
boolean	disableParallelParse
static int	GUESS_COL_CNT
static int	GUESS_HEADER
static byte	GUESS_SEP
static int	HAS_HEADER
static int	NO_HEADER

Constructor Summary

Constructors

Constructor and Description
ParseSetup() Create a default ParseSetup Used by Ray's schema magic
ParseSetup(ParserInfo parseType, byte sep, boolean singleQuotes, int checkHeader, int ncols, java.lang.String[][] data, ParseWriter.ParseErr[] errs) Create a ParseSetup without any column information Typically used by file type parsers for returning final invalid results
ParseSetup(ParserInfo parseType, byte sep, boolean singleQuotes, int checkHeader, int ncols, java.lang.String[] columnNames, byte[] ctypes, java.lang.String[][] domains, java.lang.String[][] naStrings, java.lang.String[][] data) Create a ParseSetup with all parameters except chunk size.
ParseSetup(ParserInfo parseType, byte sep, boolean singleQuotes, int checkHeader, int ncols, java.lang.String[] columnNames, byte[] ctypes, java.lang.String[][] domains, java.lang.String[][] naStrings, java.lang.String[][] data, ParseWriter.ParseErr[] errs)
ParseSetup(ParserInfo parse_type, byte sep, boolean singleQuotes, int checkHeader, int ncols, java.lang.String[] columnNames, byte[] ctypes, java.lang.String[][] domains, java.lang.String[][] naStrings, java.lang.String[][] data, ParseWriter.ParseErr[] errs, int chunkSize)
ParseSetup(ParseSetup ps)
ParseSetup(ParseSetupV3 ps) Create a ParseSetup with parameters from the client.

Method Summary

Methods

Modifier and Type	Method and Description
<T extends ParseSetup> T	copyTo(T setup) Copies the common setup to another object (that is possibly and extension of the base setup).
static java.lang.String	createHexName(java.lang.String n) Cleans up the file name to make .hex name to be used as a destination key.
java.lang.String[]	getColumnNames()
byte[]	getColumnTypes()
java.lang.String[]	getColumnTypeStrings()
java.lang.String[][]	getData()
ParseSetup	getFinalSetup(Key[] inputKeys, ParseSetup demandedSetup) Return create a final parser-specific setup for this configuration.
ParserInfo	getParseType()
static ParseSetup	guessSetup(ByteVec bv, byte[] bits, ParserInfo parserType, byte sep, int ncols, boolean singleQuotes, int checkHeader, java.lang.String[] columnNames, byte[] columnTypes, java.lang.String[][] domains, java.lang.String[][] naStrings)
static ParseSetup	guessSetup(ByteVec bv, byte[] bits, ParseSetup userSetup) Guess everything from a single pile-o-bits.
static ParseSetup	guessSetup(Key[] fkeys, boolean singleQuote, int checkHeader) Used by test harnesses for simple parsing of test data.
static ParseSetup	guessSetup(Key[] fkeys, ParseSetup userSetup) Discover the parse setup needed to correctly parse all files.
protected boolean	isCompatible(ParseSetup setupB)
static ParseSetup	makeSVMLightSetup()
protected Parser	parser(Key jobKey) This is a single entry-point to create a parser.
ParseSetup	setCheckHeader(int check_header)
ParseSetup	setChunkSize(int chunk_size)
ParseSetup	setColumnNames(java.lang.String[] column_names)
ParseSetup	setColumnTypes(byte[] column_types)
ParseSetup	setDomains(java.lang.String[][] domains)
void	setFileName(java.lang.String name)
ParseSetup	setNAStrings(java.lang.String[][] na_strings)
ParseSetup	setNumberColumns(int number_columns)
ParseSetup	setParseType(ParserInfo parse_type)
ParseSetup	setSeparator(byte separator)
ParseSetup	setSingleQuotes(boolean single_quotes)
static byte[]	strToColumnTypes(java.lang.String[] strs)
java.lang.String	toString()

Methods inherited from class water.Iced

asBytes, clone, copyOver, frozenType, read, readExternal, readJSON, reloadFromBytes, toJsonString, write, writeExternal, writeJSON

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

GUESS_SEP

public static final byte GUESS_SEP

See Also:

Constant Field Values

NO_HEADER

public static final int NO_HEADER

See Also:

Constant Field Values

GUESS_HEADER

public static final int GUESS_HEADER

See Also:

Constant Field Values

HAS_HEADER

public static final int HAS_HEADER

See Also:

Constant Field Values

GUESS_COL_CNT

public static final int GUESS_COL_CNT

See Also:

Constant Field Values

disableParallelParse

public boolean disableParallelParse

_errs

public ParseWriter.ParseErr[] _errs

_chunk_size

public int _chunk_size

Constructor Detail

ParseSetup

public ParseSetup(ParseSetup ps)

ParseSetup

public ParseSetup(ParserInfo parse_type,
          byte sep,
          boolean singleQuotes,
          int checkHeader,
          int ncols,
          java.lang.String[] columnNames,
          byte[] ctypes,
          java.lang.String[][] domains,
          java.lang.String[][] naStrings,
          java.lang.String[][] data,
          ParseWriter.ParseErr[] errs,
          int chunkSize)

ParseSetup

public ParseSetup(ParseSetupV3 ps)

Create a ParseSetup with parameters from the client. Typically used to guide sampling in the data to verify chosen settings, and fill in missing settings.

Parameters:

ps - Parse setup settings from client

ParseSetup

public ParseSetup(ParserInfo parseType,
          byte sep,
          boolean singleQuotes,
          int checkHeader,
          int ncols,
          java.lang.String[] columnNames,
          byte[] ctypes,
          java.lang.String[][] domains,
          java.lang.String[][] naStrings,
          java.lang.String[][] data)

Create a ParseSetup with all parameters except chunk size. Typically used by file type parsers for returning final valid results _chunk_size will be set later using results from all files.

ParseSetup

public ParseSetup(ParserInfo parseType,
          byte sep,
          boolean singleQuotes,
          int checkHeader,
          int ncols,
          java.lang.String[] columnNames,
          byte[] ctypes,
          java.lang.String[][] domains,
          java.lang.String[][] naStrings,
          java.lang.String[][] data,
          ParseWriter.ParseErr[] errs)

ParseSetup

public ParseSetup(ParserInfo parseType,
          byte sep,
          boolean singleQuotes,
          int checkHeader,
          int ncols,
          java.lang.String[][] data,
          ParseWriter.ParseErr[] errs)

Create a ParseSetup without any column information Typically used by file type parsers for returning final invalid results

ParseSetup

public ParseSetup()

Create a default ParseSetup Used by Ray's schema magic

Method Detail

setFileName

public void setFileName(java.lang.String name)

makeSVMLightSetup

public static ParseSetup makeSVMLightSetup()

getColumnNames

public java.lang.String[] getColumnNames()

getData

public java.lang.String[][] getData()

getColumnTypeStrings

public java.lang.String[] getColumnTypeStrings()

getColumnTypes

public byte[] getColumnTypes()

strToColumnTypes

public static byte[] strToColumnTypes(java.lang.String[] strs)

parser

protected Parser parser(Key jobKey)

This is a single entry-point to create a parser. Should be override in subclasses.

getFinalSetup

public final ParseSetup getFinalSetup(Key[] inputKeys,
                       ParseSetup demandedSetup)

Return create a final parser-specific setup for this configuration.

Parameters:

inputKeys - inputs

demandedSetup - setup demanded by a user

Returns:

a parser specific setup based on demanded setup

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object

guessSetup

public static ParseSetup guessSetup(Key[] fkeys,
                    boolean singleQuote,
                    int checkHeader)

Used by test harnesses for simple parsing of test data. Presumes auto-detection for file and separator types.

Parameters:

fkeys - Keys to input vectors to be parsed

singleQuote - single quotes quote fields

checkHeader - check for a header

Returns:

ParseSetup settings from looking at all files

guessSetup

public static ParseSetup guessSetup(Key[] fkeys,
                    ParseSetup userSetup)

Discover the parse setup needed to correctly parse all files. This takes a ParseSetup as guidance. Each file is examined individually and then results merged. If a conflict exists between any results all files are re-examined using the best guess from the first examination.

Parameters:

fkeys - Keys to input vectors to be parsed

userSetup - Setup guidance from user

Returns:

ParseSetup settings from looking at all files

isCompatible

protected boolean isCompatible(ParseSetup setupB)

guessSetup

public static ParseSetup guessSetup(ByteVec bv,
                    byte[] bits,
                    ParseSetup userSetup)

Guess everything from a single pile-o-bits. Used in tests, or in initial parser inspections when the user has not told us anything about separators or headers.

Parameters:

bits - Initial bytes from a parse source

Returns:

ParseSetup settings from looking at all files

guessSetup

public static ParseSetup guessSetup(ByteVec bv,
                    byte[] bits,
                    ParserInfo parserType,
                    byte sep,
                    int ncols,
                    boolean singleQuotes,
                    int checkHeader,
                    java.lang.String[] columnNames,
                    byte[] columnTypes,
                    java.lang.String[][] domains,
                    java.lang.String[][] naStrings)

createHexName

public static java.lang.String createHexName(java.lang.String n)

Cleans up the file name to make .hex name to be used as a destination key. Eliminates common file extensions, and replaces odd characters.

Parameters:

n - filename to be cleaned

Returns:

cleaned name

copyTo

public <T extends ParseSetup> T copyTo(T setup)

Copies the common setup to another object (that is possibly and extension of the base setup). Note: this method only copies fields directly declared in ParseSetup class, it doesn't handle fields that are declared in classes derived from ParseSetup.

Type Parameters:

T - class derived from ParseSetup

Parameters:

setup - target setup object

Returns:

the target setup object (for convenience)

getParseType

public ParserInfo getParseType()

setParseType

public ParseSetup setParseType(ParserInfo parse_type)

setSeparator

public ParseSetup setSeparator(byte separator)

setSingleQuotes

public ParseSetup setSingleQuotes(boolean single_quotes)

setCheckHeader

public ParseSetup setCheckHeader(int check_header)

setNumberColumns

public ParseSetup setNumberColumns(int number_columns)

setColumnNames

public ParseSetup setColumnNames(java.lang.String[] column_names)

setColumnTypes

public ParseSetup setColumnTypes(byte[] column_types)

setDomains

public ParseSetup setDomains(java.lang.String[][] domains)

setNAStrings

public ParseSetup setNAStrings(java.lang.String[][] na_strings)

setChunkSize

public ParseSetup setChunkSize(int chunk_size)