Spark Frame <–> H2O Frame Conversions¶
Quick links:
Converting an H2OFrame into an RDD[T]¶
The H2OContext class provides the method asRDD, which creates an RDD-like wrapper around the provided H2O’s H2OFrame:
def asRDD[A <: Product: TypeTag: ClassTag](fr: H2OFrame): RDD[A]
The call expects the type A to create a correctly typed RDD. The conversion requires type A to be bound by the Product interface.
The relationship between the columns of the H2OFrame and the attributes of class A is based on name matching.
Example¶
case class Person(name: String, age: Int)
val rdd = asRDD[Person](h2oFrame)
Converting an H2OFrame into a DataFrame¶
The H2OContext class provides the method asSparkFrame, which creates a DataFrame-like wrapper around the provided H2OFrame:
def asSparkFrame(fr: H2OFrame): DataFrame
The schema of the created instance of the DataFrame is derived from the column names and types of the specified H2OFrame.
Example¶
val dataFrame = asSparkFrame(h2oFrame)
Converting an RDD[T] into an H2OFrame¶
The H2OContext provides a conversion method from the specified RDD[A] to H2OFrame. As with conversion
in the opposite direction, the type A has to satisfy the upper bound expressed by the type Product. The conversion
creates a new H2OFrame, transfers data from the specified RDD, and saves it to the DKV store on the H2O backend.
def asH2OFrame[A <: Product : TypeTag](rdd : RDD[A]): H2OFrame
The API also provides a version, which allows for specifying the name for the resulting H2OFrame.
def asH2OFrame[A <: Product : TypeTag](rdd : RDD[A], frameName: String): H2OFrame
Example¶
val h2oFrame = h2oContext.asH2OFrame(rdd)
Converting a DataFrame into an H2OFrame¶
The H2OContext provides conversion method from the specified DataFrame to H2OFrame.
The conversion creates a new H2OFrame, transfers data from the specified DataFrame, and saves it
to the DKV store on the H2O backend.
def asH2OFrame(df: DataFrame): H2OFrame
The API also provides a version, which allows for specifying the name for the resulting H2OFrame.
def asH2OFrame(rdd : DataFrame, frameName: String): H2OFrame
Example¶
val h2oFrame = h2oContext.asH2OFrame(df)