Train Isolation Forest Model in Sparkling Water¶
Sparkling Water provides API for H2O Isolation Forest in Scala and Python. The following sections describe how to train the Isolation Forest model in Sparkling Water in both languages. See also Parameters of H2OIsolationForest.
- Scala
- Python
First, let’s start Sparkling Shell as
./bin/sparkling-shell
Start H2O cluster inside the Spark environment
import ai.h2o.sparkling._
import java.net.URI
val hc = H2OContext.getOrCreate()
Parse the data using H2O and convert them to Spark Frame
import org.apache.spark.SparkFiles
spark.sparkContext.addFile("https://raw.githubusercontent.com/h2oai/sparkling-water/master/examples/smalldata/anomaly/ecg_discord_train.csv")
spark.sparkContext.addFile("https://raw.githubusercontent.com/h2oai/sparkling-water/master/examples/smalldata/anomaly/ecg_discord_test.csv")
val trainingDF = spark.read.option("header", "true").option("inferSchema", "true").csv(SparkFiles.get("ecg_discord_train.csv"))
val testingDF = spark.read.option("header", "true").option("inferSchema", "true").csv(SparkFiles.get("ecg_discord_test.csv"))
Train the model. You can configure all the available Isolation Forest arguments using provided setters.
import ai.h2o.sparkling.ml.algos.H2OIsolationForest
val estimator = new H2OIsolationForest()
val model = estimator.fit(trainingDF)
You can also get raw model details by calling the getModelDetails() method available on the model as:
model.getModelDetails()
Run Predictions
model.transform(testingDF).show(false)
Train Isolation Forest with H2OGridSearch¶
If you’re not sure about exact values for hyper-parameters of Isolation Forest, you can plug H2OIsolationForest
to
H2OGridSearch
and define a hyper-parameter space to be walked through. Unlike other Sparkling Water algorithms used in
H2OGridSearch
, you must pass validationDataFrame
to H2OIsolationForest
as a parameter in order to
H2OGridSearch
be able to evaluate produced models. The validation data frame has to have an extra column identifying
whether the row represents an anomaly or not. The column can contain only two string values, where a value for the negative
case, must be alphabetically smaller then the value for the positive case. E.g.: "0"
/"1"
, "no"
/"yes"
,
"false"
/"true"
, etc.
- Scala
- Python
Let’s load a training and validation dataset at first:
import org.apache.spark.SparkFiles
spark.sparkContext.addFile("https://raw.githubusercontent.com/h2oai/sparkling-water/master/examples/smalldata/prostate/prostate.csv")
spark.sparkContext.addFile("https://raw.githubusercontent.com/h2oai/sparkling-water/master/examples/smalldata/prostate/prostate_anomaly_validation.csv")
val trainingDF = spark.read.option("header", "true").option("inferSchema", "true").csv(SparkFiles.get("prostate.csv"))
val validationDF = spark.read.option("header", "true").option("inferSchema", "true").csv(SparkFiles.get("prostate_anomaly_validation.csv"))
Create an algorithm instance, pass validation data frame, and specify a column identifying an anomaly:
import ai.h2o.sparkling.ml.algos.H2OIsolationForest
val algorithm = new H2OIsolationForest()
algorithm.setValidationDataFrame(validationDF)
algorithm.setValidationLabelCol("isAnomaly")
Define a hyper-parameter space:
import scala.collection.mutable
val hyperParams: mutable.HashMap[String, Array[AnyRef]] = mutable.HashMap()
hyperParams += "ntrees" -> Array(10, 20, 30).map(_.asInstanceOf[AnyRef])
hyperParams += "maxDepth" -> Array(5, 10, 20).map(_.asInstanceOf[AnyRef])
Pass the prepared hyper-parameter space and algorithm to H2OGridSearch
and run it:
import ai.h2o.sparkling.ml.algos.H2OGridSearch
val grid = new H2OGridSearch()
grid.setAlgo(algorithm)
grid.setHyperParameters(hyperParams)
val model = grid.fit(trainingDF)
Logloss
is a default metric for the model comparision produced by grids and can be changed via the method
setSelectBestModelBy
on H2OGridSearch
.