Using H2O Binary Model in Sparkling Water

Sparkling Water can generate binary models and can also load already existing models trained for example in H2O-3.

Train Model in Sparkling Water and Obtain Binary model

We first train a model using Sparkling Water API from which we can extract the binary model class. The binary model class contains methods used to work with binary models.

  • Scala
  • Python
import ai.h2o.sparkling._
val hc = H2OContext.getOrCreate()

Parse the data using H2O and convert them to Spark Frame

import org.apache.spark.SparkFiles
spark.sparkContext.addFile("https://raw.githubusercontent.com/h2oai/sparkling-water/master/examples/smalldata/prostate/prostate.csv")
val sparkDF = spark.read.option("header", "true").option("inferSchema", "true").csv(SparkFiles.get("prostate.csv"))

Select algorithm, set the parameter keepBinaryModels to true, train the model.

import ai.h2o.sparkling.ml.algos.H2OXGBoostClassifier
val estimator = new H2OXGBoostClassifier().setLabelCol("CAPSULE").setKeepBinaryModels(true)
val mojoModel = estimator.fit(sparkDF)

To obtain the binary model once the model training has finished, run:

  • Scala
  • Python
estimator.getBinaryModel()

You can store the model on disk as:

  • Scala
  • Python
val binaryModel = estimator.getBinaryModel()
binaryModel.write("/tmp/binary.model")

The loaded end exported models are always equal to each other.

Load existing binary Model

Before you start, please make sure that your H2OContext is running as we need H2O to be running. Also please make sure that Sparkling Water is of the same version as the H2O version in which the binary model was trained. If this condition is not met, Sparkling Water throws an exception.

To load binary model, run:

  • Scala
  • Python
  • R
import ai.h2o.sparkling._
import ai.h2o.sparkling.ml.models.H2OBinaryModel
val hc = H2OContext.getOrCreate()
val model = H2OBinaryModel.read(path)