H2OContext

Value Members

final def !=(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def !=(arg0: Any): Boolean

Definition Classes
Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def ==(arg0: Any): Boolean

Definition Classes
Any
val _conf: H2OConf

H2O and Spark configuration
def asDataFrame(s: String, copyMetadata: Boolean)(implicit sqlContext: SQLContext): DataFrame
def asDataFrame[T <: Frame](fr: T, copyMetadata: Boolean = true)(implicit sqlContext: SQLContext): DataFrame
def asH2OFrame(fr: Frame): H2OFrame

Create a new H2OFrame based on existing Frame
def asH2OFrame(s: String): H2OFrame

Create a new H2OFrame based on existing Frame referenced by its key.
def asH2OFrame[T <: Product](ds: Dataset[T], frameName: String)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): H2OFrame
def asH2OFrame[T <: Product](ds: Dataset[T], frameName: Option[String])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): H2OFrame
def asH2OFrame[T <: Product](ds: Dataset[T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): H2OFrame

Transforms Dataset[Supported type] to H2OFrame
def asH2OFrame(df: DataFrame, frameName: String): H2OFrame
def asH2OFrame(df: DataFrame, frameName: Option[String]): H2OFrame
def asH2OFrame(df: DataFrame): H2OFrame

Transform DataFrame to H2OFrame
def asH2OFrame(rdd: SupportedRDD, frameName: String): H2OFrame
def asH2OFrame(rdd: SupportedRDD, frameName: Option[String]): H2OFrame
def asH2OFrame(rdd: SupportedRDD): H2OFrame

Transforms RDD[Supported type] to H2OFrame
final def asInstanceOf[T0]: T0

Definition Classes
Any
def asRDD[A <: Product](implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A], arg1: ClassTag[A]): AnyRef { def apply[T <: org.apache.spark.h2o.Frame](fr: T): org.apache.spark.h2o.converters.H2ORDD[A,T] }

A generic convert of Frame into Product RDD type
A generic convert of Frame into Product RDD type
Consider using asH2OFrame since asRDD has several limitations such as that asRDD can't be used in Spark REPL in case we are RDD[T] where T is class defined in REPL. This is because class T is created as inner class and we are not able to create instance of class T without outer scope - which is impossible to get.
This code: hc.asRDD[PUBDEV458Type](rdd) will need to be call as hc.asRDD[PUBDEV458Type].apply(rdd)
def asRDD[A <: Product](fr: H2OFrame)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A], arg1: ClassTag[A]): RDD[A]

Convert given H2O frame into a Product RDD type
Convert given H2O frame into a Product RDD type
Consider using asH2OFrame since asRDD has several limitations such as that asRDD can't be used in Spark REPL in case we are RDD[T] where T is class defined in REPL. This is because class T is created as inner class and we are not able to create instance of class T without outer scope - which is impossible to get.
def buildSparkMajorVersion: String

Returns Major Spark version for which is this version of Sparkling Water designated.
Returns Major Spark version for which is this version of Sparkling Water designated.
For example, for 1.6.1 returns 1.6

Definition Classes
H2OContextUtils
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
def convert[T](hc: H2OContext, rdd: RDD[T], keyName: String, colNames: Array[String], vecTypes: Array[Byte], func: (String, Array[Byte], Option[Map[Int, NodeDesc]]) ⇒ (TaskContext, Iterator[T]) ⇒ (Int, Long)): water.fvec.H2OFrame

Converts the RDD to H2O Frame using specified conversion function
Converts the RDD to H2O Frame using specified conversion function
T
type of RDD to convert
hc
H2O context
rdd
rdd to convert
keyName
key of the resulting frame
colNames
names of the columns in the H2O Frame
vecTypes
types of the vectors in the H2O Frame
func
conversion function - the function takes parameters needed extra by specific transformations and returns function which does the general transformation
returns
H2O Frame

Definition Classes
ConverterUtils
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
def finalizeFrame[T](keyName: String, res: Array[Long], colTypes: Array[Byte], colDomains: Array[Array[String]] = null): Frame

Definition Classes
ConverterUtils
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def getConf: H2OConf

Return a copy of this H2OContext's configuration.
Return a copy of this H2OContext's configuration. The configuration cannot be changed at runtime.
def getFrameOrNone(keyName: String): Option[H2OFrame]

Gets frame for specified key or none if that frame does not exist
Gets frame for specified key or none if that frame does not exist
keyName
key of the requested frame
returns
option containing frame or none

Definition Classes
ConverterUtils
def h2oLocalClient: String
def h2oLocalClientIp: String
def h2oLocalClientPort: Int
def hashCode(): Int

Definition Classes
AnyRef → Any
object implicits extends H2OContextImplicits with Serializable

Define implicits available via h2oContext.
def init(): H2OContext

This method connects to external H2O cluster if spark.
This method connects to external H2O cluster if spark.ext.h2o.externalClusterMode is set to true, otherwise it creates new H2O cluster living in Spark
def initFrame[T](keyName: String, names: Array[String]): Unit

Definition Classes
ConverterUtils
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def isRunningOnCorrectSpark(sc: SparkContext): Boolean

Checks whether version of provided Spark is the same as Spark's version designated for this Sparkling Water version.
Checks whether version of provided Spark is the same as Spark's version designated for this Sparkling Water version. We check for correct version in shell scripts and during the build but we need to do the check also in the code in cases when the user executes for example spark-shell command with sparkling water assembly jar passed as --jars and initiates H2OContext. (Because in that case no check for correct Spark version has been done so far.)

Definition Classes
H2OContextUtils
def isTesting(sc: SparkContext): Boolean

Return true if running inside spark/sparkling water test.
Return true if running inside spark/sparkling water test.
sc
Spark Context
returns
true if the actual run is test run

Definition Classes
H2OContextUtils
def isTraceEnabled(): Boolean

Attributes
protected
Definition Classes
Logging
def log: Logger

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logName: String

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def openFlow(): Unit

Open H2O Flow running in this client.
def openURI(sc: SparkContext, uri: String): Unit

Open browser for given address.
Open browser for given address.
uri
address to open in browser, e.g., http://example.com

Definition Classes
H2OContextUtils
def setH2OClientLogLevel(level: String): Unit
def setH2ONodeLogLevel(level: String): Unit
val sparkContext: SparkContext

Spark Context
val sqlc: SQLContext
def stop(stopSparkContext: Boolean = false): Unit

Stops H2O context.
Stops H2O context.
stopSparkContext
stop also spark context
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toDataFrame[T <: water.fvec.Frame](hc: H2OContext, fr: T, copyMetadata: Boolean)(implicit sqlContext: SQLContext): DataFrame

Create a Spark DataFrame from given H2O frame.
Create a Spark DataFrame from given H2O frame.
T
type of H2O frame
hc
an instance of H2O context
fr
an instance of H2O frame
copyMetadata
copy H2O metadata into Spark DataFrame
sqlContext
running sqlContext
returns
a new DataFrame definition using given H2OFrame as data source

Definition Classes
SparkDataFrameConverter
def toH2OFrame(hc: H2OContext, rdd: SupportedRDD, frameKeyName: Option[String]): H2OFrame

Transform supported type for conversion to H2OFrame
Transform supported type for conversion to H2OFrame

Definition Classes
SupportedRDDConverter
def toH2OFrame(hc: H2OContext, dataFrame: DataFrame, frameKeyName: Option[String]): water.fvec.H2OFrame

Transform Spark's DataFrame into H2O Frame
Transform Spark's DataFrame into H2O Frame

Definition Classes
SparkDataFrameConverter
def toH2OFrameKey(df: DataFrame, frameName: String): Key[Frame]
def toH2OFrameKey(df: DataFrame, frameName: Option[String]): Key[Frame]
def toH2OFrameKey(df: DataFrame): Key[Frame]

Transform DataFrame to H2OFrame key
def toH2OFrameKey(rdd: SupportedRDD, frameName: String): Key[_]
def toH2OFrameKey(rdd: SupportedRDD, frameName: Option[String]): Key[_]
def toH2OFrameKey(rdd: SupportedRDD): Key[_]

Transforms RDD[Supported type] to H2OFrame key
def toRDD[A <: Product, T <: Frame](hc: H2OContext, fr: T)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A], arg1: ClassTag[A]): H2ORDD[A, T]

Transform H2OFrame to RDD
Transform H2OFrame to RDD

Definition Classes
SupportedRDDConverter
def toString(): String

Open Spark task manager.
Open Spark task manager.

Definition Classes
H2OContext → AnyRef → Any
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Deprecated Value Members

def asSchemaRDD[T <: Frame](fr: T, copyMetadata: Boolean = true)(implicit sqlContext: SQLContext): DataFrame

Convert given H2O frame into DataFrame type
Convert given H2O frame into DataFrame type

Annotations
@deprecated
Deprecated
(Since version Use asDataFrame) 1.3

class H2OContext extends Logging with Serializable with SparkDataFrameConverter with SupportedRDDConverter with H2OContextUtils

Value Members

final def !=(arg0: AnyRef): Boolean

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: AnyRef): Boolean

final def ==(arg0: Any): Boolean

val _conf: H2OConf

def asDataFrame(s: String, copyMetadata: Boolean)(implicit sqlContext: SQLContext): DataFrame

def asDataFrame[T <: Frame](fr: T, copyMetadata: Boolean = true)(implicit sqlContext: SQLContext): DataFrame

def asH2OFrame(fr: Frame): H2OFrame

def asH2OFrame(s: String): H2OFrame

def asH2OFrame[T <: Product](ds: Dataset[T], frameName: String)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): H2OFrame

def asH2OFrame[T <: Product](ds: Dataset[T], frameName: Option[String])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): H2OFrame

def asH2OFrame[T <: Product](ds: Dataset[T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): H2OFrame

def asH2OFrame(df: DataFrame, frameName: String): H2OFrame

def asH2OFrame(df: DataFrame, frameName: Option[String]): H2OFrame

def asH2OFrame(df: DataFrame): H2OFrame

def asH2OFrame(rdd: SupportedRDD, frameName: String): H2OFrame

def asH2OFrame(rdd: SupportedRDD, frameName: Option[String]): H2OFrame

def asH2OFrame(rdd: SupportedRDD): H2OFrame

final def asInstanceOf[T0]: T0

def asRDD[A <: Product](implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A], arg1: ClassTag[A]): AnyRef { def apply[T <: org.apache.spark.h2o.Frame](fr: T): org.apache.spark.h2o.converters.H2ORDD[A,T] }

def asRDD[A <: Product](fr: H2OFrame)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A], arg1: ClassTag[A]): RDD[A]

def buildSparkMajorVersion: String

def clone(): AnyRef

def convert[T](hc: H2OContext, rdd: RDD[T], keyName: String, colNames: Array[String], vecTypes: Array[Byte], func: (String, Array[Byte], Option[Map[Int, NodeDesc]]) ⇒ (TaskContext, Iterator[T]) ⇒ (Int, Long)): water.fvec.H2OFrame

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

def finalizeFrame[T](keyName: String, res: Array[Long], colTypes: Array[Byte], colDomains: Array[Array[String]] = null): Frame

final def getClass(): Class[_]

def getConf: H2OConf

def getFrameOrNone(keyName: String): Option[H2OFrame]

def h2oLocalClient: String

def h2oLocalClientIp: String

def h2oLocalClientPort: Int

def hashCode(): Int

object implicits extends H2OContextImplicits with Serializable

def init(): H2OContext

def initFrame[T](keyName: String, names: Array[String]): Unit

final def isInstanceOf[T0]: Boolean

def isRunningOnCorrectSpark(sc: SparkContext): Boolean

def isTesting(sc: SparkContext): Boolean

def isTraceEnabled(): Boolean

def log: Logger

def logDebug(msg: ⇒ String, throwable: Throwable): Unit

def logDebug(msg: ⇒ String): Unit

def logError(msg: ⇒ String, throwable: Throwable): Unit

def logError(msg: ⇒ String): Unit

def logInfo(msg: ⇒ String, throwable: Throwable): Unit

def logInfo(msg: ⇒ String): Unit

def logName: String

def logTrace(msg: ⇒ String, throwable: Throwable): Unit

def logTrace(msg: ⇒ String): Unit

def logWarning(msg: ⇒ String, throwable: Throwable): Unit

def logWarning(msg: ⇒ String): Unit

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def openFlow(): Unit

def openURI(sc: SparkContext, uri: String): Unit

def setH2OClientLogLevel(level: String): Unit

def setH2ONodeLogLevel(level: String): Unit

val sparkContext: SparkContext

val sqlc: SQLContext

def stop(stopSparkContext: Boolean = false): Unit

final def synchronized[T0](arg0: ⇒ T0): T0

def toDataFrame[T <: water.fvec.Frame](hc: H2OContext, fr: T, copyMetadata: Boolean)(implicit sqlContext: SQLContext): DataFrame

def toH2OFrame(hc: H2OContext, rdd: SupportedRDD, frameKeyName: Option[String]): H2OFrame

def toH2OFrame(hc: H2OContext, dataFrame: DataFrame, frameKeyName: Option[String]): water.fvec.H2OFrame

def toH2OFrameKey(df: DataFrame, frameName: String): Key[Frame]

def toH2OFrameKey(df: DataFrame, frameName: Option[String]): Key[Frame]

def toH2OFrameKey(df: DataFrame): Key[Frame]

def toH2OFrameKey(rdd: SupportedRDD, frameName: String): Key[_]

def toH2OFrameKey(rdd: SupportedRDD, frameName: Option[String]): Key[_]

def toH2OFrameKey(rdd: SupportedRDD): Key[_]

def toRDD[A <: Product, T <: Frame](hc: H2OContext, fr: T)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A], arg1: ClassTag[A]): H2ORDD[A, T]

def toString(): String