Obtain SHAP values from MOJO model¶
You can train the pipeline in Sparkling Water and get contributions from it or you can also get contributions from raw mojo. The following two sections describe how to achieve that.
Train model pipeline & get contributions¶
Obtaining SHAP values is possible only from H2OGBM, H2OXGBoost and H2ODRF pipeline wrappers and for regression or binomial problems.
To get SHAP values(=contributions) from H2OXGBoost model, please do:
- Scala
- Python
First, let’s start Sparkling Shell as
./bin/sparkling-shell
Start H2O cluster inside the Spark environment
import org.apache.spark.h2o._
import java.net.URI
val hc = H2OContext.getOrCreate(spark)
Parse the data using H2O and convert them to Spark Frame
val frame = new H2OFrame(new URI("https://raw.githubusercontent.com/h2oai/sparkling-water/master/examples/smalldata/prostate/prostate.csv"))
val sparkDF = hc.asDataFrame(frame).withColumn("CAPSULE", $"CAPSULE" cast "string")
val Array(trainingDF, testingDF) = sparkDF.randomSplit(Array(0.8, 0.2))
Train the model. You can configure all the available XGBoost arguments using provided setters, such as the label column.
import ai.h2o.sparkling.ml.algos.H2OXGBoost
val estimator = new H2OXGBoost().setLabelCol("CAPSULE").setWithDetailedPredictionCol(true)
val model = estimator.fit(trainingDF)
The call setWithDetailedPredictionCol(true)
tells Sparkling Water to create additional prediction column with
additional prediction details, such as the contributions. The name of this column is by default “detailed_prediction”
and can be modified via setDetailedPredictionCol
setter.
Run Predictions
val predictions = model.transform(testingDF).show(false)
Show contributions
predictions.select("detailed_prediction.contribution").show()
Get Contributions from Raw MOJO¶
Obtaining SHAP values is possible only from MOJO’s generated for GBM, XGBoost and DRF and for
regression or binomial problems. If you don’t need to train the model and just need to load existing mojo,
there is no need to start H2OContext
.
- Scala
- Python
First, let’s start Sparkling Shell as
./bin/sparkling-shell
Parse the data using Spark
val testingDF = spark.read.option("header", "true").option("inferSchema", "true").csv("/path/to/testing/dataset.csv")
Load the existing MOJO and enable generation of contributions via the settings object.
import ai.h2o.sparkling.ml.models._
val path = '/path/to/mojo.zip'
val settings = H2OMOJOSettings(withDetailedPredictionCol=true)
val model = H2OMOJOModel.createFromMojo(path, settings)
Run Predictions
val predictions = model.transform(testingDF)
Show contributions
predictions.select("detailed_prediction.contributions").show()