Metrics in H2O

H2O Model Metrics

Regression model.

copyright

3. 2016 H2O.ai

license

Apache License Version 2.0 (see LICENSE for details)

class h2o.model.metrics_base.H2OAnomalyDetectionModelMetrics(metric_json, on=None, algo='')

Bases: h2o.model.metrics_base.MetricsBase

mean_normalized_score()

Mean Normalized Anomaly Score. For Isolation Forest - normalized average path length.

Examples

>>> from h2o.estimators.isolation_forest import H2OIsolationForestEstimator
>>> train = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/anomaly/ecg_discord_train.csv")
>>> test = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/anomaly/ecg_discord_test.csv")
>>> isofor_model = H2OIsolationForestEstimator(sample_size=5, ntrees=7)
>>> isofor_model.train(training_frame = train)
>>> perf = isofor_model.model_performance()
>>> perf.mean_normalized_score()

mean_score()

Mean Anomaly Score. For Isolation Forest represents the average of all tree-path lengths.

Examples

>>> from h2o.estimators.isolation_forest import H2OIsolationForestEstimator
>>> train = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/anomaly/ecg_discord_train.csv")
>>> test = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/anomaly/ecg_discord_test.csv")
>>> isofor_model = H2OIsolationForestEstimator(sample_size=5, ntrees=7)
>>> isofor_model.train(training_frame = train)
>>> perf = isofor_model.model_performance()
>>> perf.mean_score()

class h2o.model.metrics_base.H2OAutoEncoderModelMetrics(metric_json, on=None, algo='')

Bases: h2o.model.metrics_base.MetricsBase

Examples

>>> from h2o.estimators.deeplearning import H2OAutoEncoderEstimator
>>> train_ecg = h2o.import_file("http://h2o-public-test-data.s3.amazonaws.com/smalldata/anomaly/ecg_discord_train.csv")
>>> test_ecg = h2o.import_file("http://h2o-public-test-data.s3.amazonaws.com/smalldata/anomaly/ecg_discord_test.csv")
>>> anomaly_model = H2OAutoEncoderEstimator(activation="Tanh",
...                                         hidden=[50,50,50],
...                                         sparse=True, l1=1e-4,
...                                         epochs=100)
>>> anomaly_model.train(x=train_ecg.names, training_frame=train_ecg)
>>> anomaly_model.mse()

class h2o.model.metrics_base.H2OBinomialModelMetrics(metric_json, on=None, algo='')

Bases: h2o.model.metrics_base.MetricsBase

This class is essentially an API for the AUC object. This class contains methods for inspecting the AUC for different criteria. To input the different criteria, use the static variable criteria.

F0point5(thresholds=None)

Parameters

thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

Returns

The F0.5 for this set of metrics and thresholds.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234)
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.F0point5()

F1(thresholds=None)

Parameters

thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

Returns

The F1 for the given set of thresholds.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234)
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.F1()

F2(thresholds=None)

Parameters

thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

Returns

The F2 for this set of metrics and thresholds.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234)
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.F2()

accuracy(thresholds=None)

Parameters

thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

Returns

The accuracy for this set of metrics and thresholds.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234)
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.accuracy()

confusion_matrix(metrics=None, thresholds=None)

Get the confusion matrix for the specified metric.

Parameters

• metrics – A string (or list of strings) among metrics listed in maximizing_metrics. Defaults to 'f1'.

• thresholds – A value (or list of values) between 0 and 1. If None, then the thresholds maximizing each provided metric will be used.

Returns

a list of ConfusionMatrix objects (if there are more than one to return), a single ConfusionMatrix (if there is only one), or None if thresholds are metrics scores are missing.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["cylinders"] = cars["cylinders"].asfactor()
>>> train, valid = cars.split_frame(ratios=[.8], seed=1234)
>>> response = "cylinders"
>>> distribution = "multinomial"
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> gbm = H2OGradientBoostingEstimator(nfolds=3,
...                                    distribution=distribution)
>>> gbm.train(x=predictors,
...           y = response,
...           training_frame = train,
...           validation_frame = valid)
>>> gbm.confusion_matrix(train)

error(thresholds=None)

Parameters

thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold minimizing the error will be used.

Returns

The error for this set of metrics and thresholds.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234)
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.error()

fallout(thresholds=None)

Parameters

thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

Returns

The fallout (same as False Positive Rate) for this set of metrics and thresholds.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234)
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.fallout()

find_idx_by_threshold(threshold)

Retrieve the index in this metric’s threshold list at which the given threshold is located.

Parameters

threshold – Find the index of this input threshold.

Returns

the index.

Raises

ValueError – if no such index can be found.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> local_data = [[1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'],
...               [1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'],
...               [0, 'b'],[0, 'b'],[0, 'b'],[0, 'b'],[0, 'b'],
...               [0, 'b'],[0, 'b'],[0, 'b'],[0, 'b'],[0, 'b']]
>>> h2o_data = h2o.H2OFrame(local_data)
>>> h2o_data.set_names(['response', 'predictor'])
>>> h2o_data["response"] = h2o_data["response"].asfactor()
>>> gbm = H2OGradientBoostingEstimator(ntrees=1,
...                                    distribution="bernoulli")
>>> gbm.train(x=list(range(1,h2o_data.ncol)),
...           y="response",
...           training_frame=h2o_data)
>>> perf = gbm.model_performance()
>>> perf.find_idx_by_threshold(0.45)

find_threshold_by_max_metric(metric)

Parameters

metrics – A string among the metrics listed in maximizing_metrics.

Returns

the threshold at which the given metric is maximal.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> local_data = [[1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'],
...               [1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'],
...               [0, 'b'],[0, 'b'],[0, 'b'],[0, 'b'],[0, 'b'],
...               [0, 'b'],[0, 'b'],[0, 'b'],[0, 'b'],[0, 'b']]
>>> h2o_data = h2o.H2OFrame(local_data)
>>> h2o_data.set_names(['response', 'predictor'])
>>> h2o_data["response"] = h2o_data["response"].asfactor()
>>> gbm = H2OGradientBoostingEstimator(ntrees=1,
...                                    distribution="bernoulli")
>>> gbm.train(x=list(range(1,h2o_data.ncol)),
...           y="response",
...           training_frame=h2o_data)
>>> perf = gbm.model_performance()
>>> perf.find_threshold_by_max_metric("f1")

fnr(thresholds=None)

Parameters

thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

Returns

The False Negative Rate.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234)
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.fnr()

fpr(thresholds=None)

Parameters

thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

Returns

The False Positive Rate.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234)
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.fpr()

property fprs

Return all false positive rates for all threshold values.

Returns

a list of false positive rates.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> r = cars[0].runif()
>>> train = cars[r > .2]
>>> valid = cars[r <= .2]
>>> response_col = "economy_20mpg"
>>> distribution = "bernoulli"
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> gbm = H2OGradientBoostingEstimator(nfolds=3, distribution=distribution, fold_assignment="Random")
>>> gbm.train(y=response_col, x=predictors, validation_frame=valid, training_frame=train)
>>> (fprs, tprs) = gbm.roc(train=True, valid=False, xval=False)
>>> fprs

gains_lift()

Retrieve the Gains/Lift table.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["cylinders"] = cars["cylinders"].asfactor()
>>> train, valid = cars.split_frame(ratios=[.8], seed=1234)
>>> response_col = "cylinders"
>>> distribution = "multinomial"
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> gbm = H2OGradientBoostingEstimator(nfolds=3,
...                                    distribution=distribution)
>>> gbm.train(x=predictors,
...           y = response,
...           training_frame = train,
...           validation_frame = valid)
>>> gbm.gains_lift()

gains_lift_plot(type='both', server=False, save_plot_path=None, plot=True)

Plot Gains/Lift curves.

Parameters

• type –

  one of:

  • ”both” (default)

  • ”gains”

  • ”lift”

• server – if True, generate plot inline using matplotlib’s Anti-Grain Geometry (AGG) backend.

• save_plot_path – filename to save the plot to.

• plot – True to plot curve; False to get a gains lift table.

Returns

Gains lift table + the resulting plot (can be accessed using result.figure()).

max_per_class_error(thresholds=None)

Parameters

thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold minimizing the error will be used.

Returns

Return 1 - min(per class accuracy).

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234)
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.max_per_class_error()

maximizing_metrics = ('absolute_mcc', 'accuracy', 'precision', 'f0point5', 'f1', 'f2', 'mean_per_class_accuracy', 'min_per_class_accuracy', 'tns', 'fns', 'fps', 'tps', 'tnr', 'fnr', 'fpr', 'tpr', 'fallout', 'missrate', 'recall', 'sensitivity', 'specificity')

metrics names allowed for confusion matrix

mcc(thresholds=None)

Parameters

thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

Returns

The absolute MCC (a value between 0 and 1, 0 being totally dissimilar, 1 being identical).

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234)
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.mcc()

mean_per_class_error(thresholds=None)

Parameters

thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold minimizing the error will be used.

Returns

mean per class error.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234)
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.mean_per_class_error()

metric(metric, thresholds=None)

Parameters

• metric (str) – A metric among maximizing_metrics.

• thresholds – thresholds parameter must be a number or a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used. If ‘all’, then all stored thresholds are used and returned with the matching metric.

Returns

The set of metrics for the list of thresholds. The returned list has a ‘value’ property holding only the metric value (if no threshold provided or if provided as a number), or all the metric values (if thresholds provided as a list)

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> local_data = [[1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'],
...               [1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'],
...               [0, 'b'],[0, 'b'],[0, 'b'],[0, 'b'],[0, 'b'],
...               [0, 'b'],[0, 'b'],[0, 'b'],[0, 'b'],[0, 'b']]
>>> h2o_data = h2o.H2OFrame(local_data)
>>> h2o_data.set_names(['response', 'predictor'])
>>> h2o_data["response"] = h2o_data["response"].asfactor()
>>> gbm = H2OGradientBoostingEstimator(ntrees=1,
...                                    distribution="bernoulli")
>>> gbm.train(x=list(range(1,h2o_data.ncol)),
...           y="response",
...           training_frame=h2o_data)
>>> perf = gbm.model_performance()
>>> perf.metric("tps", [perf.find_threshold_by_max_metric("f1")])[0][1]

metrics_aliases = {'fallout': 'fpr', 'missrate': 'fnr', 'recall': 'tpr', 'sensitivity': 'tpr', 'specificity': 'tnr'}

missrate(thresholds=None)

Parameters

thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

Returns

The miss rate (same as False Negative Rate).

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234)
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.missrate()

plot(type='roc', server=False, save_plot_path=None, plot=True)

Produce the desired metric plot.

Parameters

• type –

  the type of metric plot. One of (currently supported):

  • ROC curve (‘roc’)

  • Precision Recall curve (‘pr’)

  • Gains Lift curve (‘gainslift’)

• server – if True, generate plot inline using matplotlib’s Anti-Grain Geometry (AGG) backend.

• save_plot_path – filename to save the plot to.

• plot – True to plot curve; False to get a tuple of values at axis x and y of the plot (tprs and fprs for AUC, recall and precision for PR).

Returns

None or values of x and y axis of the plot + the resulting plot (can be accessed using result.figure()).

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234)
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.plot(type="roc")
>>> cars_gbm.plot(type="pr")

precision(thresholds=None)

Parameters

thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

Returns

The precision for this set of metrics and thresholds.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234)
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.precision()

recall(thresholds=None)

Parameters

thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

Returns

Recall for this set of metrics and thresholds.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234)
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.recall()

roc()

Return the coordinates of the ROC curve as a tuple containing the false positive rates as a list and true positive rates as a list.

Returns

The ROC values.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> r = cars[0].runif()
>>> train = cars[r > .2]
>>> valid = cars[r <= .2]
>>> response_col = "economy_20mpg"
>>> distribution = "bernoulli"
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> gbm = H2OGradientBoostingEstimator(nfolds=3,
...                                    distribution=distribution,
...                                    fold_assignment="Random")
>>> gbm.train(x=predictors,
...           y=response_col,
...           validation_frame=valid,
...           training_frame=train)
>>> gbm.roc(train=True,  valid=False, xval=False)

sensitivity(thresholds=None)

Parameters

thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

Returns

Sensitivity or True Positive Rate for this set of metrics and thresholds.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234)
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.sensitivity()

specificity(thresholds=None)

Parameters

thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

Returns

The specificity (same as True Negative Rate).

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234)
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.specificity()

tnr(thresholds=None)

Parameters

thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

Returns

The True Negative Rate.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234)
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.tnr()

tpr(thresholds=None)

Parameters

thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

Returns

The True Postive Rate.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234)
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.tpr()

property tprs

Return all true positive rates for all threshold values.

Returns

a list of true positive rates.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> r = cars[0].runif()
>>> train = cars[r > .2]
>>> valid = cars[r <= .2]
>>> response_col = "economy_20mpg"
>>> distribution = "bernoulli"
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> gbm = H2OGradientBoostingEstimator(nfolds=3, distribution=distribution, fold_assignment="Random")
>>> gbm.train(y=response_col, x=predictors, validation_frame=valid, training_frame=train)
>>> (fprs, tprs) = gbm.roc(train=True, valid=False, xval=False)
>>> tprs

class h2o.model.metrics_base.H2OBinomialUpliftModelMetrics(metric_json, on=None, algo='')

Bases: h2o.model.metrics_base.MetricsBase

This class is available only for Uplift DRF model. This class is essentially an API for the AUUC object.

aecu(metric='AUTO')

Retrieve AECU value (average excess cumulative uplift - area between Uplift curve and random curve).

Parameters

metric –

AECU metric type One of:

• ”None”

• ”qini”

• ”lift”

• ”gain”

• ”AUTO” (default; defaults to “qini”)

Returns

AECU value.

Examples

>>> from h2o.estimators import H2OUpliftRandomForestEstimator
>>> train = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/uplift/criteo_uplift_13k.csv")
>>> treatment_column = "treatment"
>>> response_column = "conversion"
>>> train[treatment_column] = train[treatment_column].asfactor()
>>> train[response_column] = train[response_column].asfactor()
>>> predictors = ["f1", "f2", "f3", "f4", "f5", "f6"]
>>>
>>> uplift_model = H2OUpliftRandomForestEstimator(ntrees=10, 
...                                               max_depth=5,
...                                               treatment_column=treatment_column,
...                                               uplift_metric="kl",
...                                               distribution="bernoulli",
...                                               min_rows=10,
...                                               auuc_type="gain")
>>> uplift_model.train(y=response_column, x=predictors, training_frame=train)
>>> perf = uplift_model.model_performance()
>>> perf.aecu()

aecu_table()

Retrieve all types of AECU values in a table.

Returns

a table of AECU values.

Examples

>>> from h2o.estimators import H2OUpliftRandomForestEstimator
>>> train = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/uplift/criteo_uplift_13k.csv")
>>> treatment_column = "treatment"
>>> response_column = "conversion"
>>> train[treatment_column] = train[treatment_column].asfactor()
>>> train[response_column] = train[response_column].asfactor()
>>> predictors = ["f1", "f2", "f3", "f4", "f5", "f6"]
>>>
>>> uplift_model = H2OUpliftRandomForestEstimator(ntrees=10, 
...                                               max_depth=5,
...                                               treatment_column=treatment_column,
...                                               uplift_metric="kl",
...                                               distribution="bernoulli",
...                                               min_rows=10,
...                                               auuc_type="gain")
>>> uplift_model.train(y=response_column, x=predictors, training_frame=train)
>>> perf = uplift_model.model_performance()
>>> perf.aecu_table()

auuc(metric=None)

Retrieve area under cumulative uplift curve (AUUC) value.

Parameters

metric –

AUUC metric type. One of:

• ”None” (default; takes default metric from model parameters)

• ”AUTO” (defaults to “qini”)

• ”qini”

• ”lift”

• ”gain”

Returns

AUUC value.

Examples

>>> from h2o.estimators import H2OUpliftRandomForestEstimator
>>> train = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/uplift/criteo_uplift_13k.csv")
>>> treatment_column = "treatment"
>>> response_column = "conversion"
>>> train[treatment_column] = train[treatment_column].asfactor()
>>> train[response_column] = train[response_column].asfactor()
>>> predictors = ["f1", "f2", "f3", "f4", "f5", "f6"]
>>>
>>> uplift_model = H2OUpliftRandomForestEstimator(ntrees=10, 
...                                               max_depth=5,
...                                               treatment_column=treatment_column,
...                                               uplift_metric="kl",
...                                               distribution="bernoulli",
...                                               min_rows=10,
...                                               auuc_type="gain")
>>> uplift_model.train(y=response_column, x=predictors, training_frame=train)
>>> perf = uplift_model.model_performance()
>>> perf.auuc()

auuc_normalized(metric=None)

Retrieve normalized area under cumulative uplift curve (AUUC) value.

Parameters

metric –

AUUC metric type. One of:

• ”None” (default; takes default metric from model parameters)

• ”AUTO” (defaults to “qini”)

• ”qini”

• ”lift”

• ”gain”

Returns

normalized AUUC value.

Examples

>>> from h2o.estimators import H2OUpliftRandomForestEstimator
>>> train = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/uplift/criteo_uplift_13k.csv")
>>> treatment_column = "treatment"
>>> response_column = "conversion"
>>> train[treatment_column] = train[treatment_column].asfactor()
>>> train[response_column] = train[response_column].asfactor()
>>> predictors = ["f1", "f2", "f3", "f4", "f5", "f6"]
>>>
>>> uplift_model = H2OUpliftRandomForestEstimator(ntrees=10, 
...                                               max_depth=5,
...                                               treatment_column=treatment_column,
...                                               uplift_metric="kl",
...                                               distribution="bernoulli",
...                                               min_rows=10,
...                                               auuc_type="gain")
>>> uplift_model.train(y=response_column, x=predictors, training_frame=train)
>>> perf = uplift_model.model_performance()
>>> perf.auuc_normalized()

auuc_table()

Retrieve all types of AUUC in a table.

Returns

a table of AUUCs.

Examples

>>> from h2o.estimators import H2OUpliftRandomForestEstimator
>>> train = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/uplift/criteo_uplift_13k.csv")
>>> treatment_column = "treatment"
>>> response_column = "conversion"
>>> train[treatment_column] = train[treatment_column].asfactor()
>>> train[response_column] = train[response_column].asfactor()
>>> predictors = ["f1", "f2", "f3", "f4", "f5", "f6"]
>>>
>>> uplift_model = H2OUpliftRandomForestEstimator(ntrees=10, 
...                                               max_depth=5,
...                                               treatment_column=treatment_column,
...                                               uplift_metric="kl",
...                                               distribution="bernoulli",
...                                               min_rows=10,
...                                               auuc_type="gain")
>>> uplift_model.train(y=response_column, x=predictors, training_frame=train)
>>> perf = uplift_model.model_performance()
>>> perf.auuc_table()

n()

Retrieve cumulative sum of numbers of observations in each bin.

Returns

a list of numbers of observation.

Examples

>>> from h2o.estimators import H2OUpliftRandomForestEstimator
>>> train = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/uplift/criteo_uplift_13k.csv")
>>> treatment_column = "treatment"
>>> response_column = "conversion"
>>> train[treatment_column] = train[treatment_column].asfactor()
>>> train[response_column] = train[response_column].asfactor()
>>> predictors = ["f1", "f2", "f3", "f4", "f5", "f6"]
>>>
>>> uplift_model = H2OUpliftRandomForestEstimator(ntrees=10, 
...                                               max_depth=5,
...                                               treatment_column=treatment_column,
...                                               uplift_metric="kl",
...                                               distribution="bernoulli",
...                                               min_rows=10,
...                                               auuc_type="gain")
>>> uplift_model.train(y=response_column, x=predictors, training_frame=train)
>>> perf = uplift_model.model_performance()
>>> perf.n()

plot_uplift(server=False, save_to_file=None, plot=True, metric='AUTO', normalize=False)

Plot Uplift Curve.

Parameters

• server – if True, generate plot inline using matplotlib’s Anti-Grain Geometry (AGG) backend.

• save_to_file – filename to save the plot to.

• plot – True to plot curve, False to get a tuple of values at axis x and y of the plot (number of observations and uplift values)

• metric –

  AUUC metric type. One of:

  • ”qini”

  • ”lift”

  • ”gain”

  • ”AUTO” (default; defaults to “qini”)

• normalize – If True, normalized values are plotted.

Examples

>>> from h2o.estimators import H2OUpliftRandomForestEstimator
>>> train = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/uplift/criteo_uplift_13k.csv")
>>> treatment_column = "treatment"
>>> response_column = "conversion"
>>> train[treatment_column] = train[treatment_column].asfactor()
>>> train[response_column] = train[response_column].asfactor()
>>> predictors = ["f1", "f2", "f3", "f4", "f5", "f6"]
>>>
>>> uplift_model = H2OUpliftRandomForestEstimator(ntrees=10, 
...                                               max_depth=5,
...                                               treatment_column=treatment_column,
...                                               uplift_metric="kl",
...                                               distribution="bernoulli",
...                                               min_rows=10,
...                                               auuc_type="gain")
>>> uplift_model.train(y=response_column, x=predictors, training_frame=train)
>>> perf = uplift_model.model_performance()
>>> perf.plot_uplift(plot=True)
>>> n, uplift = perf.plot_uplift(plot=False)

qini()

Retrieve Qini value (area between Qini cumulative uplift curve and random curve).

Returns

Qini value.

Examples

>>> from h2o.estimators import H2OUpliftRandomForestEstimator
>>> train = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/uplift/criteo_uplift_13k.csv")
>>> treatment_column = "treatment"
>>> response_column = "conversion"
>>> train[treatment_column] = train[treatment_column].asfactor()
>>> train[response_column] = train[response_column].asfactor()
>>> predictors = ["f1", "f2", "f3", "f4", "f5", "f6"]
>>>
>>> uplift_model = H2OUpliftRandomForestEstimator(ntrees=10, 
...                                               max_depth=5,
...                                               treatment_column=treatment_column,
...                                               uplift_metric="kl",
...                                               distribution="bernoulli",
...                                               min_rows=10,
...                                               auuc_type="gain")
>>> uplift_model.train(y=response_column, x=predictors, training_frame=train)
>>> perf = uplift_model.model_performance()
>>> perf.qini()

thresholds()

Retrieve prediction thresholds for each bin.

Returns

a list of thresholds.

Examples

>>> from h2o.estimators import H2OUpliftRandomForestEstimator
>>> train = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/uplift/criteo_uplift_13k.csv")
>>> treatment_column = "treatment"
>>> response_column = "conversion"
>>> train[treatment_column] = train[treatment_column].asfactor()
>>> train[response_column] = train[response_column].asfactor()
>>> predictors = ["f1", "f2", "f3", "f4", "f5", "f6"]
>>>
>>> uplift_model = H2OUpliftRandomForestEstimator(ntrees=10, 
...                                               max_depth=5,
...                                               treatment_column=treatment_column,
...                                               uplift_metric="kl",
...                                               distribution="bernoulli",
...                                               min_rows=10,
...                                               auuc_type="gain")
>>> uplift_model.train(y=response_column, x=predictors, training_frame=train)
>>> perf = uplift_model.model_performance()
>>> perf.thresholds()

thresholds_and_metric_scores()

Retrieve thresholds and metric scores table.

Returns

a thresholds and metric scores table for the specified key(s).

Examples

>>> from h2o.estimators import H2OUpliftRandomForestEstimator
>>> train = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/uplift/criteo_uplift_13k.csv")
>>> treatment_column = "treatment"
>>> response_column = "conversion"
>>> train[treatment_column] = train[treatment_column].asfactor()
>>> train[response_column] = train[response_column].asfactor()
>>> predictors = ["f1", "f2", "f3", "f4", "f5", "f6"]
>>>
>>> uplift_model = H2OUpliftRandomForestEstimator(ntrees=10, 
...                                               max_depth=5,
...                                               treatment_column=treatment_column,
...                                               uplift_metric="kl",
...                                               distribution="bernoulli",
...                                               min_rows=10,
...                                               auuc_type="gain")
>>> uplift_model.train(y=response_column, x=predictors, training_frame=train)
>>> perf = uplift_model.model_performance()
>>> perf.thresholds_and_metric_scores()

uplift(metric='AUTO')

Retrieve uplift values for each bin.

Parameters

metric –

AUUC metric type. One of:

• ”qini”

• ”lift”

• ”gain”

• ”AUTO” (default; defaults to “qini”)

Returns

a list of uplift values.

Examples

>>> from h2o.estimators import H2OUpliftRandomForestEstimator
>>> train = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/uplift/criteo_uplift_13k.csv")
>>> treatment_column = "treatment"
>>> response_column = "conversion"
>>> train[treatment_column] = train[treatment_column].asfactor()
>>> train[response_column] = train[response_column].asfactor()
>>> predictors = ["f1", "f2", "f3", "f4", "f5", "f6"]
>>>
>>> uplift_model = H2OUpliftRandomForestEstimator(ntrees=10, 
...                                               max_depth=5,
...                                               treatment_column=treatment_column,
...                                               uplift_metric="kl",
...                                               distribution="bernoulli",
...                                               min_rows=10,
...                                               auuc_type="gain")
>>> uplift_model.train(y=response_column, x=predictors, training_frame=train)
>>> perf = uplift_model.model_performance()
>>> perf.uplift()

uplift_normalized(metric='AUTO')

Retrieve normalized uplift values for each bin.

Parameters

metric –

AUUC metric type. One of:

• ”qini”

• ”lift”

• ”gain”

• ”AUTO” (default; defaults to “qini”)

Returns

a list of normalized uplift values.

Examples

>>> from h2o.estimators import H2OUpliftRandomForestEstimator
>>> train = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/uplift/criteo_uplift_13k.csv")
>>> treatment_column = "treatment"
>>> response_column = "conversion"
>>> train[treatment_column] = train[treatment_column].asfactor()
>>> train[response_column] = train[response_column].asfactor()
>>> predictors = ["f1", "f2", "f3", "f4", "f5", "f6"]
>>>
>>> uplift_model = H2OUpliftRandomForestEstimator(ntrees=10, 
...                                               max_depth=5,
...                                               treatment_column=treatment_column,
...                                               uplift_metric="kl",
...                                               distribution="bernoulli",
...                                               min_rows=10,
...                                               auuc_type="gain")
>>> uplift_model.train(y=response_column, x=predictors, training_frame=train)
>>> perf = uplift_model.model_performance()
>>> perf.uplift_normalized()

uplift_random(metric='AUTO')

Retrieve random uplift values for each bin.

Parameters

metric –

AUUC metric type. One of:

• ”qini”

• ”lift”

• ”gain”

• ”AUTO” (default; defaults to “qini”)

Returns

a list of random uplift values.

Examples

>>> from h2o.estimators import H2OUpliftRandomForestEstimator
>>> train = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/uplift/criteo_uplift_13k.csv")
>>> treatment_column = "treatment"
>>> response_column = "conversion"
>>> train[treatment_column] = train[treatment_column].asfactor()
>>> train[response_column] = train[response_column].asfactor()
>>> predictors = ["f1", "f2", "f3", "f4", "f5", "f6"]
>>>
>>> uplift_model = H2OUpliftRandomForestEstimator(ntrees=10, 
...                                               max_depth=5,
...                                               treatment_column=treatment_column,
...                                               uplift_metric="kl",
...                                               distribution="bernoulli",
...                                               min_rows=10,
...                                               auuc_type="gain")
>>> uplift_model.train(y=response_column, x=predictors, training_frame=train)
>>> perf = uplift_model.model_performance()
>>> perf.uplift_random()

class h2o.model.metrics_base.H2OClusteringModelMetrics(metric_json, on=None, algo='')

Bases: h2o.model.metrics_base.MetricsBase

betweenss()

The Between Cluster Sum-of-Square Error, or None if not present.

Examples

>>> from h2o.estimators.kmeans import H2OKMeansEstimator
>>> iris = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/iris/iris_train.csv")
>>> km = H2OKMeansEstimator(k=3, nfolds=3)
>>> km.train(x=list(range(4)), training_frame=iris)
>>> km.betweenss()

tot_withinss()

The Total Within Cluster Sum-of-Square Error, or None if not present.

Examples

>>> from h2o.estimators.kmeans import H2OKMeansEstimator
>>> iris = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/iris/iris_train.csv")
>>> km = H2OKMeansEstimator(k=3, nfolds=3)
>>> km.train(x=list(range(4)), training_frame=iris)
>>> km.tot_withinss()

totss()

The Total Sum-of-Square Error to Grand Mean, or None if not present.

Examples

>>> from h2o.estimators.kmeans import H2OKMeansEstimator
>>> iris = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/iris/iris_train.csv")
>>> km = H2OKMeansEstimator(k=3, nfolds=3)
>>> km.train(x=list(range(4)), training_frame=iris)
>>> km.totss()

class h2o.model.metrics_base.H2ODimReductionModelMetrics(metric_json, on=None, algo='')

Bases: h2o.model.metrics_base.MetricsBase

cat_err()

The Number of Misclassified categories over non-missing categorical entries, or None if not present.

num_err()

Sum of Squared Error over non-missing numeric entries, or None if not present.

class h2o.model.metrics_base.H2OHGLMModelMetrics(metric_json, on=None, algo='HGLM Gaussian Gaussian')

Bases: h2o.model.metrics_base.MetricsBase

class h2o.model.metrics_base.H2OModelMetricsRegressionCoxPH(metric_json, on=None, algo='')

Bases: h2o.model.metrics_base.MetricsBase

Examples

>>> from h2o.estimators.coxph import H2OCoxProportionalHazardsEstimator
>>> heart = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/coxph_test/heart.csv")
>>> coxph = H2OCoxProportionalHazardsEstimator(start_column="start",
...                                            stop_column="stop",
...                                            ties="breslow")
>>> coxph.train(x="age", y="event", training_frame=heart)
>>> coxph

concordance()

Concordance metrics (c-index). Proportion of concordant pairs divided by the total number of possible evaluation pairs. 1.0 for perfect match, 0.5 for random results.

concordant()

Count of concordant pairs.

tied_y()

Count of tied pairs.

class h2o.model.metrics_base.H2OMultinomialModelMetrics(metric_json, on=None, algo='')

Bases: h2o.model.metrics_base.MetricsBase

confusion_matrix()

Returns a confusion matrix based on H2O’s default prediction threshold for a dataset.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["cylinders"] = cars["cylinders"].asfactor()
>>> train, valid = cars.split_frame(ratios=[.8], seed=1234)
>>> response_col = "cylinders"
>>> distribution = "multinomial"
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> gbm = H2OGradientBoostingEstimator(nfolds=3,
...                                    distribution = distribution)
>>> gbm.train(x=predictors,
...           y = response,
...           training_frame = train,
...           validation_frame = valid)
>>> gbm.confusion_matrix(train)

hit_ratio_table()

Retrieve the Hit Ratios.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["cylinders"] = cars["cylinders"].asfactor()
>>> train, valid = cars.split_frame(ratios=[.8], seed=1234)
>>> response_col = "cylinders"
>>> distribution = "multinomial"
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> gbm = H2OGradientBoostingEstimator(nfolds=3,
...                                    distribution = distribution)
>>> gbm.train(x=predictors,
...           y = response,
...           training_frame = train,
...           validation_frame = valid)
>>> gbm.hit_ratio_table()

multinomial_auc_table()

Retrieve the multinomial AUC values.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["cylinders"] = cars["cylinders"].asfactor()
>>> train, valid = cars.split_frame(ratios=[.8], seed=1234)
>>> response_col = "cylinders"
>>> distribution = "multinomial"
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> gbm = H2OGradientBoostingEstimator(nfolds=3,
...                                    distribution = distribution)
>>> gbm.train(x=predictors,
...           y = response,
...           training_frame = train,
...           validation_frame = valid)
>>> gbm.multinomial_auc_table()

multinomial_aucpr_table()

Retrieve the multinomial PR AUC values.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["cylinders"] = cars["cylinders"].asfactor()
>>> train, valid = cars.split_frame(ratios=[.8], seed=1234)
>>> response_col = "cylinders"
>>> distribution = "multinomial"
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> gbm = H2OGradientBoostingEstimator(nfolds=3,
...                                    distribution = distribution)
>>> gbm.train(x=predictors,
...           y = response,
...           training_frame = train,
...           validation_frame = valid)
>>> gbm.multinomial_aucpr_table()

class h2o.model.metrics_base.H2OOrdinalModelMetrics(metric_json, on=None, algo='')

Bases: h2o.model.metrics_base.MetricsBase

confusion_matrix()

Returns a confusion matrix based of H2O’s default prediction threshold for a dataset.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["cylinders"] = cars["cylinders"].asfactor()
>>> train, valid = cars.split_frame(ratios=[.8], seed=1234)
>>> response_col = "cylinders"
>>> distribution = "multinomial"
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> gbm = H2OGradientBoostingEstimator(nfolds=3,
...                                    distribution = distribution)
>>> gbm.train(x=predictors,
...           y = response,
...           training_frame = train,
...           validation_frame = valid)
>>> gbm.confusion_matrix(train)

hit_ratio_table()

Retrieve the Hit Ratios.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["cylinders"] = cars["cylinders"].asfactor()
>>> train, valid = cars.split_frame(ratios=[.8], seed=1234)
>>> response_col = "cylinders"
>>> distribution = "multinomial"
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> gbm = H2OGradientBoostingEstimator(nfolds=3,
...                                    distribution = distribution)
>>> gbm.train(x=predictors,
...           y = response,
...           training_frame = train,
...           validation_frame = valid)
>>> gbm.hit_ratio_table()

class h2o.model.metrics_base.H2ORegressionModelMetrics(metric_json, on=None, algo='')

Bases: h2o.model.metrics_base.MetricsBase

This class provides an API for inspecting the metrics returned by a regression model.

It is possible to retrieve the \(R^2\) (1 - MSE/variance) and MSE.

Examples

>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "cylinders"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_glm = H2OGeneralizedLinearEstimator()
>>> cars_glm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_glm.mse()

class h2o.model.metrics_base.H2OTargetEncoderMetrics(metric_json, on=None, algo='')

Bases: h2o.model.metrics_base.MetricsBase

class h2o.model.metrics_base.H2OWordEmbeddingModelMetrics(metric_json, on=None, algo='')

Bases: h2o.model.metrics_base.MetricsBase

class h2o.model.metrics_base.List

Bases: list

class h2o.model.metrics_base.MetricsBase(metric_json, on=None, algo='')

Bases: h2o.model.metrics_base.MetricsBase

A parent class to house common metrics available for the various Metrics types.

The methods here are available across different model categories.

aic()

The AIC for this set of metrics.

Examples

>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> prostate = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/prostate/prostate.csv.zip")
>>> prostate[2] = prostate[2].asfactor()
>>> prostate[4] = prostate[4].asfactor()
>>> prostate[5] = prostate[5].asfactor()
>>> prostate[8] = prostate[8].asfactor()
>>> predictors = ["AGE","RACE","DPROS","DCAPS","PSA","VOL","GLEASON"]
>>> response = "CAPSULE"
>>> train, valid = prostate.split_frame(ratios=[.8],seed=1234)
>>> pros_glm = H2OGeneralizedLinearEstimator(family="binomial")
>>> pros_glm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> pros_glm.aic()

auc()

The AUC for this set of metrics.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) 
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.auc()

aucpr()

The area under the precision recall curve.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) 
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.aucpr()

custom_metric_name()

Name of custom metric or None.

custom_metric_value()

Value of custom metric or None.

gini()

Gini coefficient.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) 
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.gini()

hglm_metric(metric_string)

logloss()

Log loss.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) 
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.logloss()

mae()

The MAE for this set of metrics.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "cylinders"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(distribution = "poisson",
...                                         seed = 1234)
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.mae()

classmethod make(kvs)

Factory method to instantiate a MetricsBase object from the list of key-value pairs.

mean_per_class_error()

The mean per class error.

Examples

>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> prostate = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/prostate/prostate.csv.zip")
>>> prostate[2] = prostate[2].asfactor()
>>> prostate[4] = prostate[4].asfactor()
>>> prostate[5] = prostate[5].asfactor()
>>> prostate[8] = prostate[8].asfactor()
>>> predictors = ["AGE","RACE","DPROS","DCAPS","PSA","VOL","GLEASON"]
>>> response = "CAPSULE"
>>> train, valid = prostate.split_frame(ratios=[.8],seed=1234)
>>> pros_glm = H2OGeneralizedLinearEstimator(family="binomial")
>>> pros_glm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> pros_glm.mean_per_class_error()

mean_residual_deviance()

The mean residual deviance for this set of metrics.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> airlines= h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/airlines/AirlinesTest.csv.zip")
>>> air_gbm = H2OGradientBoostingEstimator()
>>> air_gbm.train(x=list(range(9)),
...               y=9,
...               training_frame=airlines,
...               validation_frame=airlines)
>>> air_gbm.mean_residual_deviance(train=True,valid=False,xval=False)

mse()

The MSE for this set of metrics.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) 
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.mse()

nobs()

The number of observations.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) 
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> perf = cars_gbm.model_performance()
>>> perf.nobs()

null_degrees_of_freedom()

The null DoF if the model has residual deviance, otherwise None.

Examples

>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> prostate = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/prostate/prostate.csv.zip")
>>> prostate[2] = prostate[2].asfactor()
>>> prostate[4] = prostate[4].asfactor()
>>> prostate[5] = prostate[5].asfactor()
>>> prostate[8] = prostate[8].asfactor()
>>> predictors = ["AGE","RACE","DPROS","DCAPS","PSA","VOL","GLEASON"]
>>> response = "CAPSULE"
>>> train, valid = prostate.split_frame(ratios=[.8],seed=1234)
>>> pros_glm = H2OGeneralizedLinearEstimator(family="binomial")
>>> pros_glm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> pros_glm.null_degrees_of_freedom()

null_deviance()

The null deviance if the model has residual deviance, otherwise None.

Examples

>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> prostate = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/prostate/prostate.csv.zip")
>>> prostate[2] = prostate[2].asfactor()
>>> prostate[4] = prostate[4].asfactor()
>>> prostate[5] = prostate[5].asfactor()
>>> prostate[8] = prostate[8].asfactor()
>>> predictors = ["AGE","RACE","DPROS","DCAPS","PSA","VOL","GLEASON"]
>>> response = "CAPSULE"
>>> train, valid = prostate.split_frame(ratios=[.8],seed=1234)
>>> pros_glm = H2OGeneralizedLinearEstimator(family="binomial")
>>> pros_glm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> pros_glm.null_deviance()

pr_auc()

MetricsBase.pr_auc is deprecated, please use MetricsBase.aucpr instead.

r2()

The R squared coefficient.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) 
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.r2()

residual_degrees_of_freedom()

The residual DoF if the model has residual deviance, otherwise None.

Examples

>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> prostate = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/prostate/prostate.csv.zip")
>>> prostate[2] = prostate[2].asfactor()
>>> prostate[4] = prostate[4].asfactor()
>>> prostate[5] = prostate[5].asfactor()
>>> prostate[8] = prostate[8].asfactor()
>>> predictors = ["AGE","RACE","DPROS","DCAPS","PSA","VOL","GLEASON"]
>>> response = "CAPSULE"
>>> train, valid = prostate.split_frame(ratios=[.8],seed=1234)
>>> pros_glm = H2OGeneralizedLinearEstimator(family="binomial")
>>> pros_glm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> pros_glm.residual_degrees_of_freedom()

residual_deviance()

The residual deviance if the model has it, otherwise None.

Examples

>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> prostate = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/prostate/prostate.csv.zip")
>>> prostate[2] = prostate[2].asfactor()
>>> prostate[4] = prostate[4].asfactor()
>>> prostate[5] = prostate[5].asfactor()
>>> prostate[8] = prostate[8].asfactor()
>>> predictors = ["AGE","RACE","DPROS","DCAPS","PSA","VOL","GLEASON"]
>>> response = "CAPSULE"
>>> train, valid = prostate.split_frame(ratios=[.8],seed=1234)
>>> pros_glm = H2OGeneralizedLinearEstimator(family="binomial")
>>> pros_glm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> pros_glm.residual_deviance()

rmse()

The RMSE for this set of metrics.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) 
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.rmse()

rmsle()

The RMSLE for this set of metrics.

Examples

>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "cylinders"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(distribution = "poisson",
...                                         seed = 1234)
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.rmsle()

show()

Display a short summary of the metrics.

Examples

>>> from from h2o.estimators.gbm import H2OGradientBoostingEstimator
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) 
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.show()

H2O Grid Metrics

class h2o.grid.metrics.H2OAutoEncoderGridSearch

Bases: object

anomaly(test_data, per_feature=False)

Obtain the reconstruction error for the input test_data.

Parameters

• test_data (H2OFrame) – The dataset upon which the reconstruction error is computed.

• per_feature (bool) – Whether to return the square reconstruction error per feature. Otherwise, return the mean square error.

Returns

the reconstruction error.

Example

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators import H2OAutoEncoderEstimator
>>> rows = [[1,2,3,4,0]*50,
...         [2,1,2,4,1]*50,
...         [2,1,4,2,1]*50,
...         [0,1,2,34,1]*50,
...         [2,3,4,1,0]*50]
>>> fr = h2o.H2OFrame(rows)
>>> hyper_parameters = {'activation': "Tanh", 'hidden': [50,50,50]}
>>> gs = H2OGridSearch(H2OAutoEncoderEstimator(), hyper_parameters)
>>> gs.train(x=range(4), training_frame=fr)
>>> gs.anomaly(fr, per_feature=True)

class h2o.grid.metrics.H2OBinomialGridSearch

Bases: object

F0point5(thresholds=None, train=False, valid=False, xval=False)

Get the F0.5 for a set of thresholds.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

• train (bool) – If train is True, then return the F0point5 value for the training data.

• valid (bool) – If valid is True, then return the F0point5 value for the validation data.

• xval (bool) – If xval is True, then return the F0point5 value for the cross validation data.

Returns

The F0point5 for this binomial model.

Examples

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> training_data = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/logreg/benign.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family='binomial'),
...                                                  hyper_parameters)
>>> gs.train(x=[3, 4-11],
...          y=3,
...          training_frame=training_data)
>>> gs.F0point5(train=True)

F1(thresholds=None, train=False, valid=False, xval=False)

Get the F1 values for a set of thresholds for the models explored.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

• train (bool) – If True, return the F1 value for the training data.

• valid (bool) – If True, return the F1 value for the validation data.

• xval (bool) – If True, return the F1 value for each of the cross-validated splits.

Returns

Dictionary of model keys to F1 values

Examples

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> training_data = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/logreg/benign.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family='binomial'),
...                                                  hyper_parameters)
>>> gs.train(x=[3, 4-11],
...          y=3,
...          training_frame=training_data)
>>> gs.F1(train=True)

F2(thresholds=None, train=False, valid=False, xval=False)

Get the F2 for a set of thresholds.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

• train (bool) – If train is True, then return the F2 value for the training data.

• valid (bool) – If valid is True, then return the F2 value for the validation data.

• xval (bool) – If xval is True, then return the F2 value for the cross validation data.

Returns

Dictionary of model keys to F2 values.

Examples

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> training_data = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/logreg/benign.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family='binomial'),
...                                                  hyper_parameters)
>>> gs.train(x=[3, 4-11],
...          y=3,
...          training_frame=training_data)
>>> gs.F2(train=True)

accuracy(thresholds=None, train=False, valid=False, xval=False)

Get the accuracy for a set of thresholds.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

• train (bool) – If train is True, then return the accuracy value for the training data.

• valid (bool) – If valid is True, then return the accuracy value for the validation data.

• xval (bool) – If xval is True, then return the accuracy value for the cross validation data.

Returns

The accuracy for this binomial model.

Examples

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> training_data = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/logreg/benign.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family='binomial'),
...                                                  hyper_parameters)
>>> gs.train(x=[3, 4-11],
...          y=3,
...          training_frame=training_data)
>>> gs.accuracy(train=True)

confusion_matrix(metrics=None, thresholds=None, train=False, valid=False, xval=False)

Get the confusion matrix for the specified metrics/thresholds.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• metrics – A string (or list of strings) among metrics listed in H2OBinomialModelMetrics.maximizing_metrics. Defaults to 'f1'.

• thresholds – A value (or list of values) between 0 and 1. If None, then the thresholds maximizing each provided metric will be used.

• train (bool) – If train is True, then return the confusion matrix value for the training data.

• valid (bool) – If valid is True, then return the confusion matrix value for the validation data.

• xval (bool) – If xval is True, then return the confusion matrix value for the cross validation data.

Returns

The confusion matrix for this binomial model.

Examples

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> training_data = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/logreg/benign.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family='binomial'),
...                                                  hyper_parameters)
>>> gs.train(x=[3, 4-11],
...          y=3,
...          training_frame=training_data)
>>> gs.confusion_matrix(train=True)

error(thresholds=None, train=False, valid=False, xval=False)

Get the error for a set of thresholds.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold minimizing the error will be used.

• train (bool) – If train is True, then return the error value for the training data.

• valid (bool) – If valid is True, then return the error value for the validation data.

• xval (bool) – If xval is True, then return the error value for the cross validation data.

Returns

The error for this binomial model.

Examples

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> training_data = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/logreg/benign.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family='binomial'),
...                                                  hyper_parameters)
>>> gs.train(x=[3, 4-11],
...          y=3,
...          training_frame=training_data)
>>> gs.error(train=True)

fallout(thresholds=None, train=False, valid=False, xval=False)

Get the Fallout (AKA False Positive Rate) for a set of thresholds.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

• train (bool) – If train is True, then return the fallout value for the training data.

• valid (bool) – If valid is True, then return the fallout value for the validation data.

• xval (bool) – If xval is True, then return the fallout value for the cross validation data.

Returns

The fallout for this binomial model.

Examples

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> training_data = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/logreg/benign.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family='binomial'),
...                                                  hyper_parameters)
>>> gs.train(x=[3, 4-11],
...          y=3,
...          training_frame=training_data)
>>> gs.fallout(train=True)

find_idx_by_threshold(threshold, train=False, valid=False, xval=False)

Retrieve the index in this metric’s threshold list at which the given threshold is located.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• threshold (float) – The threshold value to search for.

• train (bool) – If train is True, then return the idx_by_threshold for the training data.

• valid (bool) – If valid is True, then return the idx_by_threshold for the validation data.

• xval (bool) – If xval is True, then return the idx_by_threshold for the cross validation data.

Returns

The idx_by_threshold for this binomial model.

Examples

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> training_data = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/logreg/benign.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family='binomial'),
...                                                  hyper_parameters)
>>> gs.train(x=[3, 4-11],
...          y=3,
...          training_frame=training_data)
>>> gs.find_idx_by_threshold(0.45, train=True)

find_threshold_by_max_metric(metric, train=False, valid=False, xval=False)

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• metric (str) – A metric among the metrics listed in H2OBinomialModelMetrics.maximizing_metrics.

• train (bool) – If train is True, then return the threshold_by_max_metric value for the training data.

• valid (bool) – If valid is True, then return the threshold_by_max_metric value for the validation data.

• xval (bool) – If xval is True, then return the threshold_by_max_metric value for the cross validation data.

Returns

The threshold_by_max_metric for this binomial model.

Examples

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> training_data = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/logreg/benign.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family='binomial'),
...                                                  hyper_parameters)
>>> gs.train(x=[3, 4-11],
...          y=3,
...          training_frame=training_data)
>>> gs.find_threshold_by_max_metric("tps", train=True)

fnr(thresholds=None, train=False, valid=False, xval=False)

Get the False Negative Rates for a set of thresholds. If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

• train (bool) – If train is True, then return the FNR value for the training data.

• valid (bool) – If valid is True, then return the FNR value for the validation data.

• xval (bool) – If xval is True, then return the FNR value for the cross validation data.

Returns

The FNR for this binomial model.

Examples

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> training_data = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/logreg/benign.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family='binomial'),
...                                                  hyper_parameters)
>>> gs.train(x=[3, 4-11],
...          y=3,
...          training_frame=training_data)
>>> gs.fnr(train=True)

fpr(thresholds=None, train=False, valid=False, xval=False)

Get the False Positive Rates for a set of thresholds.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

• train (bool) – If train is True, then return the FPR value for the training data.

• valid (bool) – If valid is True, then return the FPR value for the validation data.

• xval (bool) – If xval is True, then return the FPR value for the cross validation data.

Returns

The FPR for this binomial model.

Examples

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> training_data = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/logreg/benign.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family='binomial'),
...                                                  hyper_parameters)
>>> gs.train(x=[3, 4-11],
...          y=3,
...          training_frame=training_data)
>>> gs.fpr(train=True)

max_per_class_error(thresholds=None, train=False, valid=False, xval=False)

Get the max per class error for a set of thresholds.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold minimizing the error will be used.

• train (bool) – If train is True, then return the max_per_class_error value for the training data.

• valid (bool) – If valid is True, then return the max_per_class_error value for the validation data.

• xval (bool) – If xval is True, then return the max_per_class_error value for the cross validation data.

Returns

The max per class error for this binomial model.

Examples

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> training_data = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/logreg/benign.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family='binomial'),
...                                                  hyper_parameters)
>>> gs.train(x=[3, 4-11],
...          y=3,
...          training_frame=training_data)
>>> gs.max_per_class_error(train=True)

mcc(thresholds=None, train=False, valid=False, xval=False)

Get the MCC for a set of thresholds.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

• train (bool) – If train is True, then return the mcc value for the training data.

• valid (bool) – If valid is True, then return the mcc value for the validation data.

• xval (bool) – If xval is True, then return the mcc value for the cross validation data.

Returns

The MCC for this binomial model.

Examples

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> training_data = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/logreg/benign.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family='binomial'),
...                                                  hyper_parameters)
>>> gs.train(x=[3, 4-11],
...          y=3,
...          training_frame=training_data)
>>> gs.mcc(train=True)

mean_per_class_error(thresholds=None, train=False, valid=False, xval=False)

Get the mean per class error for a set of thresholds.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold minimizing the error will be used.

• train (bool) – If train is True, then return the mean_per_class_error value for the training data.

• valid (bool) – If valid is True, then return the mean_per_class_error value for the validation data.

• xval (bool) – If xval is True, then return the mean_per_class_error value for the cross validation data.

Returns

The mean per class error for this binomial model.

Examples

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> training_data = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/logreg/benign.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family='binomial'),
...                                                  hyper_parameters)
>>> gs.train(x=[3, 4-11],
...          y=3,
...          training_frame=training_data)
>>> gs.mean_per_class_error(train=True)

metric(metric, thresholds=None, train=False, valid=False, xval=False)

Get the metric value for a set of thresholds.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• metric – name of the metric to compute.

• thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

• train (bool) – If train is True, then return the metrics for the training data.

• valid (bool) – If valid is True, then return the metrics for the validation data.

• xval (bool) – If xval is True, then return the metrics for the cross validation data.

Returns

The metrics for this binomial model.

Examples

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> training_data = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/logreg/benign.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family='binomial'),
...                                                  hyper_parameters)
>>> gs.train(x=[3, 4-11],
...          y=3,
...          training_frame=training_data)
>>> gs.metric("tps", train=True)

missrate(thresholds=None, train=False, valid=False, xval=False)

Get the miss rate (AKA False Negative Rate) for a set of thresholds.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

• train (bool) – If train is True, then return the missrate value for the training data.

• valid (bool) – If valid is True, then return the missrate value for the validation data.

• xval (bool) – If xval is True, then return the missrate value for the cross validation data.

Returns

The missrate for this binomial model.

Examples

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> training_data = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/logreg/benign.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family='binomial'),
...                                                  hyper_parameters)
>>> gs.train(x=[3, 4-11],
...          y=3,
...          training_frame=training_data)
>>> gs.missrate(train=True)

precision(thresholds=None, train=False, valid=False, xval=False)

Get the precision for a set of thresholds.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

• train (bool) – If train is True, then return the precision value for the training data.

• valid (bool) – If valid is True, then return the precision value for the validation data.

• xval (bool) – If xval is True, then return the precision value for the cross validation data.

Returns

The precision for this binomial model.

Examples

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> training_data = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/logreg/benign.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family='binomial'),
...                                                  hyper_parameters)
>>> gs.train(x=[3, 4-11],
...          y=3,
...          training_frame=training_data)
>>> gs. precision(train=True)

recall(thresholds=None, train=False, valid=False, xval=False)

Get the Recall (AKA True Positive Rate) for a set of thresholds.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

• train (bool) – If train is True, then return the recall value for the training data.

• valid (bool) – If valid is True, then return the recall value for the validation data.

• xval (bool) – If xval is True, then return the recall value for the cross validation data.

Returns

The recall for this binomial model.

Examples

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> training_data = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/logreg/benign.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family='binomial'),
...                                                  hyper_parameters)
>>> gs.train(x=[3, 4-11],
...          y=3,
...          training_frame=training_data)
>>> gs.recall(train=True)

roc(train=False, valid=False, xval=False)

Return the coordinates of the ROC curve for a given set of data, as a two-tuple containing the false positive rates as a list and true positive rates as a list.

If all are False (default), then return the training data. If more than one ROC curve is requested, the data is returned as a dictionary of two-tuples.

Parameters

• train (bool) – If train is True, then return the ROC coordinates for the training data.

• valid (bool) – If valid is True, then return the ROC coordinates for the validation data.

• xval (bool) – If xval is True, then return the ROC coordinates for the cross validation data.

Returns

the true cooridinates of the roc curve.

Examples

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> training_data = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/logreg/benign.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family='binomial'),
...                                                  hyper_parameters)
>>> gs.train(x=[3, 4-11],
...          y=3,
...          training_frame=training_data)
>>> gs.roc(train=True)

sensitivity(thresholds=None, train=False, valid=False, xval=False)

Get the sensitivity (AKA True Positive Rate or Recall) for a set of thresholds.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

• train (bool) – If train is True, then return the sensitivity value for the training data.

• valid (bool) – If valid is True, then return the sensitivity value for the validation data.

• xval (bool) – If xval is True, then return the sensitivity value for the cross validation data.

Returns

The sensitivity for this binomial model.

Examples

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> training_data = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/logreg/benign.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family='binomial'),
...                                                  hyper_parameters)
>>> gs.train(x=[3, 4-11],
...          y=3,
...          training_frame=training_data)
>>> gs.sensitivity(train=True)

specificity(thresholds=None, train=False, valid=False, xval=False)

Get the specificity (AKA True Negative Rate) for a set of thresholds.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

• train (bool) – If train is True, then return the specificity value for the training data.

• valid (bool) – If valid is True, then return the specificity value for the validation data.

• xval (bool) – If xval is True, then return the specificity value for the cross validation data.

Returns

The specificity for this binomial model.

Examples

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> training_data = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/logreg/benign.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family='binomial'),
...                                                  hyper_parameters)
>>> gs.train(x=[3, 4-11],
...          y=3,
...          training_frame=training_data)
>>> gs.specificity(train=True)

tnr(thresholds=None, train=False, valid=False, xval=False)

Get the True Negative Rate for a set of thresholds.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

• train (bool) – If train is True, then return the TNR value for the training data.

• valid (bool) – If valid is True, then return the TNR value for the validation data.

• xval (bool) – If xval is True, then return the TNR value for the cross validation data.

Returns

The TNR for this binomial model.

Examples

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> training_data = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/logreg/benign.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family='binomial'),
...                                                  hyper_parameters)
>>> gs.train(x=[3, 4-11],
...          y=3,
...          training_frame=training_data)
>>> gs.tnr(train=True)

tpr(thresholds=None, train=False, valid=False, xval=False)

Get the True Positive Rate for a set of thresholds.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• thresholds – thresholds parameter must be a list (e.g. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.

• train (bool) – If train is True, then return the TPR value for the training data.

• valid (bool) – If valid is True, then return the TPR value for the validation data.

• xval (bool) – If xval is True, then return the TPR value for the cross validation data.

Returns

The TPR for this binomial model.

Examples

>>> from h2o.grid.grid_search import H2OGridSearch
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> training_data = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/logreg/benign.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family='binomial'),
...                                                  hyper_parameters)
>>> gs.train(x=[3, 4-11],
...          y=3,
...          training_frame=training_data)
>>> gs.tpr(train=True)

class h2o.grid.metrics.H2OClusteringGridSearch

Bases: object

betweenss(train=False, valid=False, xval=False)

Get the between cluster sum of squares.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• train (bool) – If True, then return the between cluster sum of squares value for the training data.

• valid (bool) – If True, then return the between cluster sum of squares value for the validation data.

• xval (bool) – If True, then return the between cluster sum of squares value for each of the cross-validated splits.

Returns

the between cluster sum of squares values for the specified key(s).

Examples

>>> from h2o.estimators import H2OKMeansEstimator
>>> from h2o.grid.grid_search import H2OGridSearch
>>> iris = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/iris/iris_train.csv")
>>> hyper_parameters = {'k': [2,3,4], 'init': "random"}
>>> gs = H2OGridSearch(H2OKMeansEstimator(), hyper_parameters)
>>> gs.train(x=list(range(4)), training_frame=iris)
>>> gs.betweenss(train=True)

centers()

Returns the centers for the KMeans model.

Examples

>>> from h2o.estimators import H2OKMeansEstimator
>>> from h2o.grid.grid_search import H2OGridSearch
>>> iris = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/iris/iris_train.csv")
>>> hyper_parameters = {'k': [2,3,4], 'init': "random"}
>>> gs = H2OGridSearch(H2OKMeansEstimator(), hyper_parameters)
>>> gs.train(x=list(range(4)), training_frame=iris)
>>> gs.centers()

centers_std()

Returns the standardized centers for the KMeans model.

Examples

>>> from h2o.estimators import H2OKMeansEstimator
>>> from h2o.grid.grid_search import H2OGridSearch
>>> iris = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/iris/iris_train.csv")
>>> hyper_parameters = {'k': [2,3,4], 'init': "random"}
>>> gs = H2OGridSearch(H2OKMeansEstimator(), hyper_parameters)
>>> gs.train(x=list(range(4)), training_frame=iris)
>>> gs.centers_std()

centroid_stats(train=False, valid=False, xval=False)

Get the centroid statistics for each cluster.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• train (bool) – If True, then return the centroid statistics for the training data.

• valid (bool) – If True, then return the centroid statistics for the validation data.

• xval (bool) – If True, then return the centroid statistics for each of the cross-validated splits.

Returns

the centroid statistics for the specified key(s).

Examples

>>> from h2o.estimators import H2OKMeansEstimator
>>> from h2o.grid.grid_search import H2OGridSearch
>>> iris = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/iris/iris_train.csv")
>>> hyper_parameters = {'k': [2,3,4], 'init': "random"}
>>> gs = H2OGridSearch(H2OKMeansEstimator(), hyper_parameters)
>>> gs.train(x=list(range(4)), training_frame=iris)
>>> gs.centroid_stats(train=True)

num_iterations()

Get the number of iterations that it took to converge or reach max iterations.

Examples

>>> from h2o.estimators import H2OKMeansEstimator
>>> from h2o.grid.grid_search import H2OGridSearch
>>> iris = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/iris/iris_train.csv")
>>> hyper_parameters = {'k': [2,3,4], 'init': "random"}
>>> gs = H2OGridSearch(H2OKMeansEstimator(), hyper_parameters)
>>> gs.train(x=list(range(4)), training_frame=iris)
>>> gs.num_iterations()

size(train=False, valid=False, xval=False)

Get the sizes of each cluster.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• train (bool) – If True, then return the cluster sizes for the training data.

• valid (bool) – If True, then return the cluster sizes for the validation data.

• xval (bool) – If True, then return the cluster sizes for each of the cross-validated splits.

Returns

the cluster sizes for the specified key(s).

Examples

>>> from h2o.estimators import H2OKMeansEstimator
>>> from h2o.grid.grid_search import H2OGridSearch
>>> iris = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/iris/iris_train.csv")
>>> hyper_parameters = {'k': [2,3,4], 'init': "random"}
>>> gs = H2OGridSearch(H2OKMeansEstimator(), hyper_parameters)
>>> gs.train(x=list(range(4)), training_frame=iris)
>>> gs.size(train=True)

tot_withinss(train=False, valid=False, xval=False)

Get the total within cluster sum of squares.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• train (bool) – If True, then return the total within cluster sum of squares for the training data.

• valid (bool) – If True, then return the total within cluster sum of squares for the validation data.

• xval (bool) – If True, then return the total within cluster sum of squares for each of the cross-validated splits.

Returns

the total within cluster sum of squares values for the specified key(s).

Examples

>>> from h2o.estimators import H2OKMeansEstimator
>>> from h2o.grid.grid_search import H2OGridSearch
>>> iris = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/iris/iris_train.csv")
>>> hyper_parameters = {'k': [2,3,4], 'init': "random"}
>>> gs = H2OGridSearch(H2OKMeansEstimator(), hyper_parameters)
>>> gs.train(x=list(range(4)), training_frame=iris)
>>> gs.tot_withinss(train=True)

totss(train=False, valid=False, xval=False)

Get the total sum of squares.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• train (bool) – If True, then return total sum of squares for the training data.

• valid (bool) – If True, then return the total sum of squares for the validation data.

• xval (bool) – If True, then return the total sum of squares for each of the cross-validated splits.

Returns

the total sum of squares values for the specified key(s).

Examples

>>> from h2o.estimators import H2OKMeansEstimator
>>> from h2o.grid.grid_search import H2OGridSearch
>>> iris = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/iris/iris_train.csv")
>>> hyper_parameters = {'k': [2,3,4], 'init': "random"}
>>> gs = H2OGridSearch(H2OKMeansEstimator(), hyper_parameters)
>>> gs.train(x=list(range(4)), training_frame=iris)
>>> gs.totss(train=True)

withinss(train=False, valid=False, xval=False)

Get the within cluster sum of squares for each cluster.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• train (bool) – If True, then return within cluster sum of squares for the training data.

• valid (bool) – If True, then return the within cluster sum of squares for the validation data.

• xval (bool) – If True, then return the within cluster sum of squares for each of the cross-validated splits.

Returns

the within cluster sum of squares values for the specified key(s).

Examples

>>> from h2o.estimators import H2OKMeansEstimator
>>> from h2o.grid.grid_search import H2OGridSearch
>>> iris = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/iris/iris_train.csv")
>>> hyper_parameters = {'k': [2,3,4], 'init': "random"}
>>> gs = H2OGridSearch(H2OKMeansEstimator(), hyper_parameters)
>>> gs.train(x=list(range(4)), training_frame=iris)
>>> gs.withinss(train=True)

class h2o.grid.metrics.H2ODimReductionGridSearch

Bases: object

archetypes()

Returns

the archetypes (Y) of the GLRM model.

Examples

>>> from h2o.estimators import H2OGeneralizedLowRankEstimator
>>> from h2o.grid.grid_search import H2OGridSearch
>>> iris = h2o.import_file("http://h2o-public-test-data.s3.amazonaws.com/smalldata/iris/iris_wheader.csv")
>>> hyper_parameters = {'gamma_x': [0.05, 0.5], 'gamma_y': [0.05,0.5]}
>>> gs = H2OGridSearch(H2OGeneralizedLowRankEstimator(),
...                    hyper_parameters)
>>> gs.train(x=iris.names, training_frame=iris)
>>> gs.archetypes()

final_step()

Get the final step size from the GLRM model.

Returns

final step size (double).

Examples

>>> from h2o.estimators import H2OGeneralizedLowRankEstimator
>>> from h2o.grid.grid_search import H2OGridSearch
>>> iris = h2o.import_file("http://h2o-public-test-data.s3.amazonaws.com/smalldata/iris/iris_wheader.csv")
>>> hyper_parameters = {'gamma_x': [0.05, 0.5], 'gamma_y': [0.05,0.5]}
>>> gs = H2OGridSearch(H2OGeneralizedLowRankEstimator(),
...                    hyper_parameters)
>>> gs.train(x=iris.names, training_frame=iris)
>>> gs.final_step()

num_iterations()

Get the number of iterations that it took to converge or reach max iterations.

Returns

number of iterations (integer).

Examples

>>> from h2o.estimators import H2OGeneralizedLowRankEstimator
>>> from h2o.grid.grid_search import H2OGridSearch
>>> iris = h2o.import_file("http://h2o-public-test-data.s3.amazonaws.com/smalldata/iris/iris_wheader.csv")
>>> hyper_parameters = {'gamma_x': [0.05, 0.5], 'gamma_y': [0.05,0.5]}
>>> gs = H2OGridSearch(H2OGeneralizedLowRankEstimator(),
...                    hyper_parameters)
>>> gs.train(x=iris.names, training_frame=iris)
>>> gs.num_iterations()

objective()

Get the final value of the objective function from the GLRM model.

Returns

final objective value (double).

Examples

>>> from h2o.estimators import H2OGeneralizedLowRankEstimator
>>> from h2o.grid.grid_search import H2OGridSearch
>>> iris = h2o.import_file("http://h2o-public-test-data.s3.amazonaws.com/smalldata/iris/iris_wheader.csv")
>>> hyper_parameters = {'gamma_x': [0.05, 0.5], 'gamma_y': [0.05,0.5]}
>>> gs = H2OGridSearch(H2OGeneralizedLowRankEstimator(),
...                    hyper_parameters)
>>> gs.train(x=iris.names, training_frame=iris)
>>> gs.objective()

class h2o.grid.metrics.H2OMultinomialGridSearch

Bases: object

auc(train=False, valid=False, xval=False)

Retrieve the AUC value.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• train (bool) – If train is True, then return the AUC values for the training data.

• valid (bool) – If valid is True, then return the AUC values for the validation data.

• xval (bool) – If xval is True, then return the AUC values for the cross validation data.

Returns

The AUC values for this multinomial model.

Examples

>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> from h2o.grid.grid_search import H2OGridSearch
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> iris = h2o.import_file("http://h2o-public-test-data.s3.amazonaws.com/smalldata/iris/iris.csv")
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family = "multinomial"),
...                                                  hyper_parameters)
>>> gs.train(x=[0,1,2,3], y=4, training_frame=iris)
>>> gs.auc(train=True)

aucpr(train=False, valid=False, xval=False)

Retrieve the PR AUC value.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• train (bool) – If train is True, then return the PR AUC values for the training data.

• valid (bool) – If valid is True, then return the PR AUC values for the validation data.

• xval (bool) – If xval is True, then return the PR AUC values for the cross validation data.

Returns

The PR AUC values for this multinomial model.

Examples

>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> from h2o.grid.grid_search import H2OGridSearch
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> iris = h2o.import_file("http://h2o-public-test-data.s3.amazonaws.com/smalldata/iris/iris.csv")
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family = "multinomial"),
...                                                  hyper_parameters)
>>> gs.train(x=[0,1,2,3], y=4, training_frame=iris)
>>> gs.aucpr(train=True)

confusion_matrix(data)

Returns a confusion matrix based of H2O’s default prediction threshold for a dataset.

Parameters

data – metric for which the confusion matrix will be calculated.

Examples

>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> from h2o.grid.grid_search import H2OGridSearch
>>> iris = h2o.import_file("http://h2o-public-test-data.s3.amazonaws.com/smalldata/iris/iris.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family = "multinomial"),
...                                                  hyper_parameters)
>>> gs.train(x=[0,1,2,3], y=4, training_frame=iris)
>>> gs.confusion_matrix(iris)

hit_ratio_table(train=False, valid=False, xval=False)

Retrieve the Hit Ratios.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• train (bool) – If train is True, then return the hit ratio value for the training data.

• valid (bool) – If valid is True, then return the hit ratio value for the validation data.

• xval (bool) – If xval is True, then return the hit ratio value for the cross validation data.

Returns

The hit ratio for this multinomial model.

Examples

>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> from h2o.grid.grid_search import H2OGridSearch
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> iris = h2o.import_file("http://h2o-public-test-data.s3.amazonaws.com/smalldata/iris/iris.csv")
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family = "multinomial"),
...                                                  hyper_parameters)
>>> gs.train(x=[0,1,2,3], y=4, training_frame=iris)
>>> gs.hit_ratio_table(train=True)

mean_per_class_error(train=False, valid=False, xval=False)

Get the mean per class error.

If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• train (bool) – If train is True, then return the mean per class error value for the training data.

• valid (bool) – If valid is True, then return the mean per class error value for the validation data.

• xval (bool) – If xval is True, then return the mean per class error value for the cross validation data.

Returns

The mean per class error for this multinomial model.

Examples

>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator
>>> from h2o.grid.grid_search import H2OGridSearch
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> iris = h2o.import_file("http://h2o-public-test-data.s3.amazonaws.com/smalldata/iris/iris.csv")
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family = "multinomial"),
...                                                  hyper_parameters)
>>> gs.train(x=[0,1,2,3], y=4, training_frame=iris)
>>> gs.mean_per_class_error(train=True)

class h2o.grid.metrics.H2OOrdinalGridSearch

Bases: object

confusion_matrix(data)

Returns a confusion matrix based of H2O’s default prediction threshold for a dataset.

Parameters

data – metric for which the confusion matrix will be calculated.

Examples

>>> from h2o.estimators import H2OGeneralizedLinearEstimator
>>> from h2o.grid.grid_search import H2OGridSearch
>>> h2o_df = h2o.import_file("http://h2o-public-test-data.s3.amazonaws.com/bigdata/laptop/glm_ordinal_logit/ordinal_multinomial_training_set.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family="ordinal"), hyper_parameters)
>>> h2o_df['C11'] = h2o_df['C11'].asfactor()
>>> gs.train(x=list(range(0,10)), y="C11", training_frame=h2o_df)
>>> gs.confusion_matrix(h2o_df)

hit_ratio_table(train=False, valid=False, xval=False)

Retrieve the Hit Ratios.

If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• train (bool) – If train is True, then return the hit ratio value for the training data.

• valid (bool) – If valid is True, then return the hit ratio value for the validation data.

• xval (bool) – If xval is True, then return the hit ratio value for the cross validation data.

Returns

The hit ratio for this ordinal model.

Examples

>>> from h2o.estimators import H2OGeneralizedLinearEstimator
>>> from h2o.grid.grid_search import H2OGridSearch
>>> h2o_df = h2o.import_file("http://h2o-public-test-data.s3.amazonaws.com/bigdata/laptop/glm_ordinal_logit/ordinal_multinomial_training_set.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family="ordinal"), hyper_parameters)
>>> h2o_df['C11'] = h2o_df['C11'].asfactor()
>>> gs.train(x=list(range(0,10)), y="C11", training_frame=h2o_df)
>>> gs.hit_ratio_table(train=True)

mean_per_class_error(train=False, valid=False, xval=False)

Get the mean per class error.

If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.

Parameters

• train (bool) – If train is True, then return the mean per class error value for the training data.

• valid (bool) – If valid is True, then return the mean per class error value for the validation data.

• xval (bool) – If xval is True, then return the mean per class error value for the cross validation data.

Returns

The mean per class error for this ordinal model.

Examples

>>> from h2o.estimators import H2OGeneralizedLinearEstimator
>>> from h2o.grid.grid_search import H2OGridSearch
>>> h2o_df = h2o.import_file("http://h2o-public-test-data.s3.amazonaws.com/bigdata/laptop/glm_ordinal_logit/ordinal_multinomial_training_set.csv")
>>> hyper_parameters = {'alpha': [0.01,0.5], 'lambda': [1e-5,1e-6]}                          
>>> gs = H2OGridSearch(H2OGeneralizedLinearEstimator(family="ordinal"), hyper_parameters)
>>> h2o_df['C11'] = h2o_df['C11'].asfactor()
>>> gs.train(x=list(range(0,10)), y="C11", training_frame=h2o_df)
>>> gs.mean_per_class_error(train=True)

class h2o.grid.metrics.H2ORegressionGridSearch

Bases: object