Class/Object

org.apache.spark.h2o

H2OContext

Related Docs: object H2OContext | package h2o

Permalink

class H2OContext extends Logging with H2OContextUtils

Create new H2OContext based on provided H2O configuration

Self Type
H2OContext
Linear Supertypes
H2OContextUtils, Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. H2OContext
  2. H2OContextUtils
  3. Logging
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. val _conf: H2OConf

    Permalink

    H2O and Spark configuration

  5. val announcementService: AnnouncementService

    Permalink
  6. def asDataFrame(s: String, copyMetadata: Boolean): DataFrame

    Permalink
  7. def asDataFrame[T <: Frame](fr: T, copyMetadata: Boolean = true): DataFrame

    Permalink

    Convert given H2O frame into DataFrame type

  8. def asH2OFrame(fr: Frame): H2OFrame

    Permalink

    Create a new H2OFrame based on existing Frame

  9. def asH2OFrame(s: String): H2OFrame

    Permalink

    Create a new H2OFrame based on existing Frame referenced by its key.

  10. def asH2OFrame[T <: Product](ds: Dataset[T], frameName: String)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): H2OFrame

    Permalink
  11. def asH2OFrame[T <: Product](ds: Dataset[T], frameName: Option[String])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): H2OFrame

    Permalink
  12. def asH2OFrame[T <: Product](ds: Dataset[T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): H2OFrame

    Permalink

    Transforms Dataset[Supported type] to H2OFrame

  13. def asH2OFrame(df: DataFrame, frameName: String): H2OFrame

    Permalink
  14. def asH2OFrame(df: DataFrame, frameName: Option[String]): H2OFrame

    Permalink
  15. def asH2OFrame(df: DataFrame): H2OFrame

    Permalink

    Transform DataFrame to H2OFrame

  16. def asH2OFrame(rdd: SupportedRDD, frameName: String): H2OFrame

    Permalink
  17. def asH2OFrame(rdd: SupportedRDD, frameName: Option[String]): H2OFrame

    Permalink
  18. def asH2OFrame(rdd: SupportedRDD): H2OFrame

    Permalink

    Transforms RDD[Supported type] to H2OFrame

  19. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  20. def asRDD[A <: Product](implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A], arg1: ClassTag[A]): AnyRef { def apply[T <: org.apache.spark.h2o.Frame](fr: T): org.apache.spark.h2o.converters.H2ORDD[A,T] }

    Permalink

    A generic convert of Frame into Product RDD type

    A generic convert of Frame into Product RDD type

    Consider using asH2OFrame since asRDD has several limitations such as that asRDD can't be used in Spark REPL in case we are RDD[T] where T is class defined in REPL. This is because class T is created as inner class and we are not able to create instance of class T without outer scope - which is impossible to get.

    This code: hc.asRDD[PUBDEV458Type](rdd) will need to be call as hc.asRDD[PUBDEV458Type].apply(rdd)

  21. def asRDD[A <: Product](fr: H2OFrame)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A], arg1: ClassTag[A]): RDD[A]

    Permalink

    Convert given H2O frame into a Product RDD type

    Convert given H2O frame into a Product RDD type

    Consider using asH2OFrame since asRDD has several limitations such as that asRDD can't be used in Spark REPL in case we are RDD[T] where T is class defined in REPL. This is because class T is created as inner class and we are not able to create instance of class T without outer scope - which is impossible to get.

  22. val backend: SparklingBackend

    Permalink

    Used backend

  23. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. def downloadH2OLogs(destination: String): String

    Permalink
    Definition Classes
    H2OContextUtils
  25. def downloadH2OLogs(destination: URI): URI

    Permalink

    destination

    directory where the logs will be downloaded

    Definition Classes
    H2OContextUtils
  26. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  27. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  28. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  29. def flowURL(): String

    Permalink
  30. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  31. def getConf: H2OConf

    Permalink

    Return a copy of this H2OContext's configuration.

    Return a copy of this H2OContext's configuration. The configuration cannot be changed at runtime.

  32. def getScheme(hc: H2OConf): String

    Permalink
    Definition Classes
    H2OContextUtils
  33. def h2oLocalClient: String

    Permalink

    Returns location of REST API of H2O client

  34. def h2oLocalClientIp: String

    Permalink

    Returns IP of H2O client

  35. def h2oLocalClientPort: Int

    Permalink

    Returns port where H2O REST API is exposed

  36. val h2oNodes: ArrayBuffer[NodeDesc]

    Permalink

    Runtime list of active H2O nodes

  37. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  38. object implicits extends H2OContextImplicits with Serializable

    Permalink

    Define implicits available via h2oContext.implicits._

  39. def importHiveTable(database: String = HiveTableImporter.DEFAULT_DATABASE, table: String, partitions: Array[Array[String]] = null, allowMultiFormat: Boolean = false): water.fvec.Frame

    Permalink
    Definition Classes
    H2OContextUtils
  40. def init(): H2OContext

    Permalink

    This method connects to external H2O cluster if spark.ext.h2o.externalClusterMode is set to true, otherwise it creates new H2O cluster living in Spark

  41. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  42. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  43. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  44. def isRunningOnCorrectSpark(sc: SparkContext): Boolean

    Permalink

    Checks whether version of provided Spark is the same as Spark's version designated for this Sparkling Water version.

    Checks whether version of provided Spark is the same as Spark's version designated for this Sparkling Water version. We check for correct version in shell scripts and during the build but we need to do the check also in the code in cases when the user executes for example spark-shell command with sparkling water assembly jar passed as --jars and initiates H2OContext. (Because in that case no check for correct Spark version has been done so far.)

    Definition Classes
    H2OContextUtils
  45. def isRunningOnDatabricks(): Boolean

    Permalink
    Definition Classes
    H2OContextUtils
  46. def isTesting(sc: SparkContext): Boolean

    Permalink

    Return true if running inside spark/sparkling water test.

    Return true if running inside spark/sparkling water test.

    sc

    Spark Context

    returns

    true if the actual run is test run

    Definition Classes
    H2OContextUtils
  47. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  48. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  49. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  50. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  51. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  52. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  53. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  54. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  55. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  56. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  57. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  58. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  59. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  60. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  61. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  62. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  63. def openFlow(): Unit

    Permalink

    Open H2O Flow running in this client.

  64. def openURI(sc: SparkContext, uri: String): Unit

    Permalink

    Open browser for given address.

    Open browser for given address.

    uri

    address to open in browser, e.g., http://example.com

    Definition Classes
    H2OContextUtils
  65. def setH2OClientLogLevel(level: String): Unit

    Permalink

    Set log level for the client running in driver

  66. def setH2OLogLevel(level: String): Unit

    Permalink

    Set H2O log level for the client and all executors

  67. def setH2ONodeLogLevel(level: String): Unit

    Permalink

    Set log level for all H2O services running on executors

  68. val sparkContext: SparkContext

    Permalink
  69. val sparkSession: SparkSession

    Permalink

    Spark Session

  70. def stop(stopSparkContext: Boolean = false): Unit

    Permalink

    Stops H2O context.

    Stops H2O context.

    stopSparkContext

    stop also spark context

  71. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  72. def toH2OFrameKey[T <: Product](ds: Dataset[T], frameName: String)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Key[Frame]

    Permalink
  73. def toH2OFrameKey[T <: Product](ds: Dataset[T], frameName: Option[String])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Key[Frame]

    Permalink
  74. def toH2OFrameKey[T <: Product](ds: Dataset[T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Key[Frame]

    Permalink

    Transforms Dataset[Supported type] to H2OFrame key

  75. def toH2OFrameKey(df: DataFrame, frameName: String): Key[Frame]

    Permalink
  76. def toH2OFrameKey(df: DataFrame, frameName: Option[String]): Key[Frame]

    Permalink
  77. def toH2OFrameKey(df: DataFrame): Key[Frame]

    Permalink

    Transform DataFrame to H2OFrame key

  78. def toH2OFrameKey(rdd: SupportedRDD, frameName: String): Key[_]

    Permalink
  79. def toH2OFrameKey(rdd: SupportedRDD, frameName: Option[String]): Key[_]

    Permalink
  80. def toH2OFrameKey(rdd: SupportedRDD): Key[_]

    Permalink

    Transforms RDD[Supported type] to H2OFrame key

  81. def toString(): String

    Permalink
    Definition Classes
    H2OContext → AnyRef → Any
  82. val uiUpdateThread: H2ORuntimeInfoUIThread

    Permalink
  83. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  84. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  85. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  86. def withConversionDebugPrints[R <: water.fvec.Frame](sc: SparkContext, conversionName: String, block: ⇒ R): R

    Permalink
    Definition Classes
    H2OContextUtils

Inherited from H2OContextUtils

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped