H2OContext

Value Members

1. final def !=(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

2. final def ##(): Int

   

   Definition Classes

   AnyRef → Any

3. final def ==(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

4. val _conf: H2OConf

   

   H2O and Spark configuration

5. def asDataFrame(s: String, copyMetadata: Boolean)(implicit sqlContext: SQLContext): DataFrame

   

6. def asDataFrame[T <: Frame](fr: T, copyMetadata: Boolean = true)(implicit sqlContext: SQLContext): DataFrame

   

   Convert given H2O frame into DataFrame type

7. def asH2OFrame(fr: Frame): H2OFrame

   

   Create a new H2OFrame based on existing Frame

8. def asH2OFrame(s: String): H2OFrame

   

   Create a new H2OFrame based on existing Frame referenced by its key.

9. def asH2OFrame[T <: Product](ds: Dataset[T], frameName: String)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): H2OFrame

   

10. def asH2OFrame[T <: Product](ds: Dataset[T], frameName: Option[String])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): H2OFrame

    

11. def asH2OFrame[T <: Product](ds: Dataset[T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): H2OFrame

    

    Transforms Dataset[Supported type] to H2OFrame

12. def asH2OFrame(df: DataFrame, frameName: String): H2OFrame

    

13. def asH2OFrame(df: DataFrame, frameName: Option[String]): H2OFrame

    

14. def asH2OFrame(df: DataFrame): H2OFrame

    

    Transform DataFrame to H2OFrame

15. def asH2OFrame(rdd: SupportedRDD, frameName: String): H2OFrame

    

16. def asH2OFrame(rdd: SupportedRDD, frameName: Option[String]): H2OFrame

    

17. def asH2OFrame(rdd: SupportedRDD): H2OFrame

    

    Transforms RDD[Supported type] to H2OFrame

18. final def asInstanceOf[T0]: T0

    

    Definition Classes

    Any

19. def asRDD[A <: Product](implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A], arg1: ClassTag[A]): AnyRef { def apply[T <: org.apache.spark.h2o.Frame](fr: T): org.apache.spark.h2o.converters.H2ORDD[A,T] }

    

    A generic convert of Frame into Product RDD type

    A generic convert of Frame into Product RDD type

    Consider using asH2OFrame since asRDD has several limitations such as that asRDD can't be used in Spark REPL in case we are RDD[T] where T is class defined in REPL. This is because class T is created as inner class and we are not able to create instance of class T without outer scope - which is impossible to get.

    This code: hc.asRDD[PUBDEV458Type](rdd) will need to be call as hc.asRDD[PUBDEV458Type].apply(rdd)

20. def asRDD[A <: Product](fr: H2OFrame)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A], arg1: ClassTag[A]): RDD[A]

    

    Convert given H2O frame into a Product RDD type

    Convert given H2O frame into a Product RDD type

    Consider using asH2OFrame since asRDD has several limitations such as that asRDD can't be used in Spark REPL in case we are RDD[T] where T is class defined in REPL. This is because class T is created as inner class and we are not able to create instance of class T without outer scope - which is impossible to get.

21. def buildSparkMajorVersion: String

    

    Returns Major Spark version for which is this version of Sparkling Water designated.

    Returns Major Spark version for which is this version of Sparkling Water designated.

    For example, for 1.6.1 returns 1.6

    Definition Classes

    H2OContextUtils

22. def clone(): AnyRef

    

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

23. def convert[T](hc: H2OContext, rdd: RDD[T], keyName: String, colNames: Array[String], vecTypes: Array[Byte], func: ConversionFunction[T]): water.fvec.H2OFrame

    

    Converts the RDD to H2O Frame using specified conversion function

    Converts the RDD to H2O Frame using specified conversion function

    T

    type of RDD to convert

    hc

    H2O context

    rdd

    rdd to convert

    keyName

    key of the resulting frame

    colNames

    names of the columns in the H2O Frame

    vecTypes

    types of the vectors in the H2O Frame

    func

    conversion function - the function takes parameters needed extra by specific transformations and returns function which does the general transformation

    returns

    H2O Frame

    Definition Classes

    ConverterUtils

24. final def eq(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

25. def equals(arg0: Any): Boolean

    

    Definition Classes

    AnyRef → Any

26. def finalize(): Unit

    

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

27. def finalizeFrame[T](keyName: String, res: Array[Long], colTypes: Array[Byte], colDomains: Array[Array[String]] = null): Frame

    

    Definition Classes

    ConverterUtils

28. final def getClass(): Class[_]

    

    Definition Classes

    AnyRef → Any

29. def getConf: H2OConf

    

    Return a copy of this H2OContext's configuration.

    Return a copy of this H2OContext's configuration. The configuration cannot be changed at runtime.

30. def getFrameOrNone(keyName: String): Option[H2OFrame]

    

    Gets frame for specified key or none if that frame does not exist

    Gets frame for specified key or none if that frame does not exist

    keyName

    key of the requested frame

    returns

    option containing frame or none

    Definition Classes

    ConverterUtils

31. def h2oLocalClient: String

    

    Returns location of REST API of H2O client

32. def h2oLocalClientIp: String

    

    Returns IP of H2O client

33. def h2oLocalClientPort: Int

    

    Returns port where H2O REST API is exposed

34. def hashCode(): Int

    

    Definition Classes

    AnyRef → Any

35. object implicits extends H2OContextImplicits with Serializable

    

    Define implicits available via h2oContext.implicits._

36. def init(): H2OContext

    

    This method connects to external H2O cluster if spark.ext.h2o.externalClusterMode is set to true, otherwise it creates new H2O cluster living in Spark

37. def initFrame[T](keyName: String, names: Array[String]): Unit

    

    Definition Classes

    ConverterUtils

38. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

39. final def isInstanceOf[T0]: Boolean

    

    Definition Classes

    Any

40. def isRunningOnCorrectSpark(sc: SparkContext): Boolean

    

    Checks whether version of provided Spark is the same as Spark's version designated for this Sparkling Water version.

    Checks whether version of provided Spark is the same as Spark's version designated for this Sparkling Water version. We check for correct version in shell scripts and during the build but we need to do the check also in the code in cases when the user executes for example spark-shell command with sparkling water assembly jar passed as --jars and initiates H2OContext. (Because in that case no check for correct Spark version has been done so far.)

    Definition Classes

    H2OContextUtils

41. def isTesting(sc: SparkContext): Boolean

    

    Return true if running inside spark/sparkling water test.

    Return true if running inside spark/sparkling water test.

    sc

    Spark Context

    returns

    true if the actual run is test run

    Definition Classes

    H2OContextUtils

42. def isTraceEnabled(): Boolean

    

    Attributes

    protected

    Definition Classes

    Logging

43. def log: Logger

    

    Attributes

    protected

    Definition Classes

    Logging

44. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

45. def logDebug(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

46. def logError(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

47. def logError(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

48. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

49. def logInfo(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

50. def logName: String

    

    Attributes

    protected

    Definition Classes

    Logging

51. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

52. def logTrace(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

53. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

54. def logWarning(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

55. final def ne(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

56. final def notify(): Unit

    

    Definition Classes

    AnyRef

57. final def notifyAll(): Unit

    

    Definition Classes

    AnyRef

58. def openFlow(): Unit

    

    Open H2O Flow running in this client.

59. def openURI(sc: SparkContext, uri: String): Unit

    

    Open browser for given address.

    Open browser for given address.

    uri

    address to open in browser, e.g., http://example.com

    Definition Classes

    H2OContextUtils

60. def setH2OClientLogLevel(level: String): Unit

    

    Set log level for the client running in driver

61. def setH2OLogLevel(level: String): Unit

    

    Set H2O log level for the client and all executors

62. def setH2ONodeLogLevel(level: String): Unit

    

    Set log level for all H2O services running on executors

63. val sparkContext: SparkContext

    

    Spark Context

64. val sqlc: SQLContext

    

65. def stop(stopSparkContext: Boolean = false): Unit

    

    Stops H2O context.

    Stops H2O context.

    stopSparkContext

    stop also spark context

66. final def synchronized[T0](arg0: ⇒ T0): T0

    

    Definition Classes

    AnyRef

67. def toDataFrame[T <: water.fvec.Frame](hc: H2OContext, fr: T, copyMetadata: Boolean)(implicit sqlContext: SQLContext): DataFrame

    

    Create a Spark DataFrame from given H2O frame.

    Create a Spark DataFrame from given H2O frame.

    T

    type of H2O frame

    hc

    an instance of H2O context

    fr

    an instance of H2O frame

    copyMetadata

    copy H2O metadata into Spark DataFrame

    sqlContext

    running sqlContext

    returns

    a new DataFrame definition using given H2OFrame as data source

    Definition Classes

    SparkDataFrameConverter

68. def toH2OFrame(hc: H2OContext, rdd: SupportedRDD, frameKeyName: Option[String]): H2OFrame

    

    Transform supported type for conversion to H2OFrame

    Transform supported type for conversion to H2OFrame

    Definition Classes

    SupportedRDDConverter

69. def toH2OFrame(hc: H2OContext, dataFrame: DataFrame, frameKeyName: Option[String]): water.fvec.H2OFrame

    

    Transform Spark's DataFrame into H2O Frame

    Transform Spark's DataFrame into H2O Frame

    Definition Classes

    SparkDataFrameConverter

70. def toH2OFrameKey(df: DataFrame, frameName: String): Key[Frame]

    

71. def toH2OFrameKey(df: DataFrame, frameName: Option[String]): Key[Frame]

    

72. def toH2OFrameKey(df: DataFrame): Key[Frame]

    

    Transform DataFrame to H2OFrame key

73. def toH2OFrameKey(rdd: SupportedRDD, frameName: String): Key[_]

    

74. def toH2OFrameKey(rdd: SupportedRDD, frameName: Option[String]): Key[_]

    

75. def toH2OFrameKey(rdd: SupportedRDD): Key[_]

    

    Transforms RDD[Supported type] to H2OFrame key

76. def toRDD[A <: Product, T <: Frame](hc: H2OContext, fr: T)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A], arg1: ClassTag[A]): H2ORDD[A, T]

    

    Transform H2OFrame to RDD

    Transform H2OFrame to RDD

    Definition Classes

    SupportedRDDConverter

77. def toString(): String

    

    Open Spark task manager.

    Open Spark task manager.

    Definition Classes

    H2OContext → AnyRef → Any

78. final def wait(): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

79. final def wait(arg0: Long, arg1: Int): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

80. final def wait(arg0: Long): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )