Sparkling Water and Zeppelin

Because Sparkling Water exposes the Scala API, it is possible to access it directly from the Zeppelin’s notebook cell marked by the %spark tag.

Launch Zeppelin with Sparkling Water

Using Sparkling Water from Zeppelin is easy because Sparkling Water is distributed as a Spark package. In this case, before launching Zeppelin, an addition shell variable is needed:

export SPARK_HOME=...# Spark 3.3 home
export SPARK_SUBMIT_OPTIONS="--packages ai.h2o:sparkling-water-package_2.12:3.46.0.1-1-3.3"
bin/zeppelin.sh -Pspark-3.3

The above command uses Spark 3.3 and the corresponding Sparkling Water package.

Using Zeppelin

The use of the Sparkling Water package is directly driven by the Sparkling Water API. For example, getting H2OContext is straightforward:

%spark
import ai.h2o.sparkling._
val hc = H2OContext.getOrCreate()

Creating an H2OFrame from a Spark DataFrame:

%spark
val df = sc.parallelize(1 to 1000).toDF
val hf = hc.asH2OFrame(df)

Creating a Spark DataFrame from an H2OFrame:

%spark
val df = hc.asSparkFrame(hf)