Metrics in H2O¶
H2O Model Metrics
¶
Regression model.
- copyright
2016 H2O.ai
- license
Apache License Version 2.0 (see LICENSE for details)
-
class
h2o.model.metrics_base.
H2OAnomalyDetectionModelMetrics
(metric_json, on=None, algo='')[source]¶ Bases:
h2o.model.metrics_base.MetricsBase
-
mean_normalized_score
()[source]¶ Mean Normalized Anomaly Score. For Isolation Forest - normalized average path length.
- Examples
>>> from h2o.estimators.isolation_forest import H2OIsolationForestEstimator >>> train = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/anomaly/ecg_discord_train.csv") >>> test = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/anomaly/ecg_discord_test.csv") >>> isofor_model = H2OIsolationForestEstimator(sample_size=5, ntrees=7) >>> isofor_model.train(training_frame = train) >>> perf = isofor_model.model_performance() >>> perf.mean_normalized_score()
-
mean_score
()[source]¶ Mean Anomaly Score. For Isolation Forest represents the average of all tree-path lengths.
- Examples
>>> from h2o.estimators.isolation_forest import H2OIsolationForestEstimator >>> train = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/anomaly/ecg_discord_train.csv") >>> test = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/anomaly/ecg_discord_test.csv") >>> isofor_model = H2OIsolationForestEstimator(sample_size=5, ntrees=7) >>> isofor_model.train(training_frame = train) >>> perf = isofor_model.model_performance() >>> perf.mean_score()
-
-
class
h2o.model.metrics_base.
H2OAutoEncoderModelMetrics
(metric_json, on=None, algo='')[source]¶ Bases:
h2o.model.metrics_base.MetricsBase
- Examples
>>> from h2o.estimators.deeplearning import H2OAutoEncoderEstimator >>> train_ecg = h2o.import_file("http://h2o-public-test-data.s3.amazonaws.com/smalldata/anomaly/ecg_discord_train.csv") >>> test_ecg = h2o.import_file("http://h2o-public-test-data.s3.amazonaws.com/smalldata/anomaly/ecg_discord_test.csv") >>> anomaly_model = H2OAutoEncoderEstimator(activation="Tanh", ... hidden=[50,50,50], ... sparse=True, l1=1e-4, ... epochs=100) >>> anomaly_model.train(x=train_ecg.names, training_frame=train_ecg) >>> anomaly_model.mse()
-
class
h2o.model.metrics_base.
H2OBinomialModelMetrics
(metric_json, on=None, algo='')[source]¶ Bases:
h2o.model.metrics_base.MetricsBase
This class is essentially an API for the AUC object. This class contains methods for inspecting the AUC for different criteria. To input the different criteria, use the static variable criteria.
-
F0point5
(thresholds=None)[source]¶ - Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
- Returns
The F0.5 for this set of metrics and thresholds.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.F0point5()
-
F1
(thresholds=None)[source]¶ - Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
- Returns
The F1 for the given set of thresholds.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.F1()
-
F2
(thresholds=None)[source]¶ - Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
- Returns
The F2 for this set of metrics and thresholds.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.F2()
-
accuracy
(thresholds=None)[source]¶ - Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
- Returns
The accuracy for this set of metrics and thresholds.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.accuracy()
-
confusion_matrix
(metrics=None, thresholds=None)[source]¶ Get the confusion matrix for the specified metric
- Parameters
metrics – A string (or list of strings) among metrics listed in
maximizing_metrics
. Defaults to ‘f1’.thresholds – A value (or list of values) between 0 and 1. If None, then the thresholds maximizing each provided metric will be used.
- Returns
a list of ConfusionMatrix objects (if there are more than one to return), or a single ConfusionMatrix (if there is only one).
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["cylinders"] = cars["cylinders"].asfactor() >>> train, valid = cars.split_frame(ratios=[.8], seed=1234) >>> response = "cylinders" >>> distribution = "multinomial" >>> predictors = ["displacement","power","weight","acceleration","year"] >>> gbm = H2OGradientBoostingEstimator(nfolds=3, ... distribution=distribution) >>> gbm.train(x=predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> gbm.confusion_matrix(train)
-
error
(thresholds=None)[source]¶ - Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold minimizing the error will be used.
- Returns
The error for this set of metrics and thresholds.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.error()
-
fallout
(thresholds=None)[source]¶ - Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
- Returns
The fallout (same as False Positive Rate) for this set of metrics and thresholds.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.fallout()
-
find_idx_by_threshold
(threshold)[source]¶ Retrieve the index in this metric’s threshold list at which the given threshold is located.
- Parameters
threshold – Find the index of this input threshold.
- Returns
the index
- Raises
ValueError – if no such index can be found.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> local_data = [[1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'], ... [1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'], ... [0, 'b'],[0, 'b'],[0, 'b'],[0, 'b'],[0, 'b'], ... [0, 'b'],[0, 'b'],[0, 'b'],[0, 'b'],[0, 'b']] >>> h2o_data = h2o.H2OFrame(local_data) >>> h2o_data.set_names(['response', 'predictor']) >>> h2o_data["response"] = h2o_data["response"].asfactor() >>> gbm = H2OGradientBoostingEstimator(ntrees=1, ... distribution="bernoulli") >>> gbm.train(x=list(range(1,h2o_data.ncol)), ... y="response", ... training_frame=h2o_data) >>> perf = gbm.model_performance() >>> perf.find_idx_by_threshold(0.45)
-
find_threshold_by_max_metric
(metric)[source]¶ - Parameters
metrics – A string among the metrics listed in
maximizing_metrics
.- Returns
the threshold at which the given metric is maximal.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> local_data = [[1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'], ... [1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'], ... [0, 'b'],[0, 'b'],[0, 'b'],[0, 'b'],[0, 'b'], ... [0, 'b'],[0, 'b'],[0, 'b'],[0, 'b'],[0, 'b']] >>> h2o_data = h2o.H2OFrame(local_data) >>> h2o_data.set_names(['response', 'predictor']) >>> h2o_data["response"] = h2o_data["response"].asfactor() >>> gbm = H2OGradientBoostingEstimator(ntrees=1, ... distribution="bernoulli") >>> gbm.train(x=list(range(1,h2o_data.ncol)), ... y="response", ... training_frame=h2o_data) >>> perf = gbm.model_performance() >>> perf.find_threshold_by_max_metric("f1")
-
fnr
(thresholds=None)[source]¶ - Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
- Returns
The False Negative Rate.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.fnr()
-
fpr
(thresholds=None)[source]¶ - Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
- Returns
The False Positive Rate.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.fpr()
-
property
fprs
¶ Return all false positive rates for all threshold values.
- Returns
a list of false positive rates.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> r = cars[0].runif() >>> train = cars[r > .2] >>> valid = cars[r <= .2] >>> response_col = "economy_20mpg" >>> distribution = "bernoulli" >>> predictors = ["displacement","power","weight","acceleration","year"] >>> gbm = H2OGradientBoostingEstimator(nfolds=3, distribution=distribution, fold_assignment="Random") >>> gbm.train(y=response_col, x=predictors, validation_frame=valid, training_frame=train) >>> (fprs, tprs) = gbm.roc(train=True, valid=False, xval=False) >>> fprs
-
gains_lift
()[source]¶ Retrieve the Gains/Lift table.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["cylinders"] = cars["cylinders"].asfactor() >>> train, valid = cars.split_frame(ratios=[.8], seed=1234) >>> response_col = "cylinders" >>> distribution = "multinomial" >>> predictors = ["displacement","power","weight","acceleration","year"] >>> gbm = H2OGradientBoostingEstimator(nfolds=3, ... distribution=distribution) >>> gbm.train(x=predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> gbm.gains_lift()
-
max_per_class_error
(thresholds=None)[source]¶ - Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold minimizing the error will be used.
- Returns
Return 1 - min(per class accuracy).
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.max_per_class_error()
-
maximizing_metrics
= ('absolute_mcc', 'accuracy', 'precision', 'f0point5', 'f1', 'f2', 'mean_per_class_accuracy', 'min_per_class_accuracy', 'tns', 'fns', 'fps', 'tps', 'tnr', 'fnr', 'fpr', 'tpr', 'missrate', 'specificity', 'fallout', 'sensitivity', 'recall')¶ metrics names allowed for confusion matrix
-
mcc
(thresholds=None)[source]¶ - Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
- Returns
The absolute MCC (a value between 0 and 1, 0 being totally dissimilar, 1 being identical).
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.mcc()
-
mean_per_class_error
(thresholds=None)[source]¶ - Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold minimizing the error will be used.
- Returns
mean per class error.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.mean_per_class_error()
-
metric
(metric, thresholds=None)[source]¶ - Parameters
metric (str) – A metric among
maximizing_metrics
.thresholds – thresholds parameter must be a number or a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used. If ‘all’, then all stored thresholds are used and returned with the matching metric.
- Returns
The set of metrics for the list of thresholds. The returned list has a ‘value’ property holding only the metric value (if no threshold provided or if provided as a number), or all the metric values (if thresholds provided as a list)
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> local_data = [[1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'], ... [1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'],[1, 'a'], ... [0, 'b'],[0, 'b'],[0, 'b'],[0, 'b'],[0, 'b'], ... [0, 'b'],[0, 'b'],[0, 'b'],[0, 'b'],[0, 'b']] >>> h2o_data = h2o.H2OFrame(local_data) >>> h2o_data.set_names(['response', 'predictor']) >>> h2o_data["response"] = h2o_data["response"].asfactor() >>> gbm = H2OGradientBoostingEstimator(ntrees=1, ... distribution="bernoulli") >>> gbm.train(x=list(range(1,h2o_data.ncol)), ... y="response", ... training_frame=h2o_data) >>> perf = gbm.model_performance() >>> perf.metric("tps", [perf.find_threshold_by_max_metric("f1")])[0][1]
-
metrics_aliases
= {'fallout': 'fpr', 'missrate': 'fnr', 'recall': 'tpr', 'sensitivity': 'tpr', 'specificity': 'tnr'}¶
-
missrate
(thresholds=None)[source]¶ - Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
- Returns
The miss rate (same as False Negative Rate).
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.missrate()
-
plot
(type='roc', server=False)[source]¶ Produce the desired metric plot.
- Parameters
type – the type of metric plot (currently, only ROC supported).
server – if True, generate plot inline using matplotlib’s “Agg” backend.
- Returns
None
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.plot()
-
precision
(thresholds=None)[source]¶ - Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
- Returns
The precision for this set of metrics and thresholds.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.precision()
-
recall
(thresholds=None)[source]¶ - Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
- Returns
Recall for this set of metrics and thresholds.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.recall()
-
roc
()[source]¶ Return the coordinates of the ROC curve as a tuple containing the false positive rates as a list and true positive rates as a list. :returns: The ROC values.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> r = cars[0].runif() >>> train = cars[r > .2] >>> valid = cars[r <= .2] >>> response_col = "economy_20mpg" >>> distribution = "bernoulli" >>> predictors = ["displacement","power","weight","acceleration","year"] >>> gbm = H2OGradientBoostingEstimator(nfolds=3, ... distribution=distribution, ... fold_assignment="Random") >>> gbm.train(x=predictors, ... y=response_col, ... validation_frame=valid, ... training_frame=train) >>> gbm.roc(train=True, valid=False, xval=False)
-
sensitivity
(thresholds=None)[source]¶ - Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
- Returns
Sensitivity or True Positive Rate for this set of metrics and thresholds.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.sensitivity()
-
specificity
(thresholds=None)[source]¶ - Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
- Returns
The specificity (same as True Negative Rate).
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.specificity()
-
tnr
(thresholds=None)[source]¶ - Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
- Returns
The True Negative Rate.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.tnr()
-
tpr
(thresholds=None)[source]¶ - Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
- Returns
The True Postive Rate.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.tpr()
-
property
tprs
¶ Return all true positive rates for all threshold values.
- Returns
a list of true positive rates.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> r = cars[0].runif() >>> train = cars[r > .2] >>> valid = cars[r <= .2] >>> response_col = "economy_20mpg" >>> distribution = "bernoulli" >>> predictors = ["displacement","power","weight","acceleration","year"] >>> gbm = H2OGradientBoostingEstimator(nfolds=3, distribution=distribution, fold_assignment="Random") >>> gbm.train(y=response_col, x=predictors, validation_frame=valid, training_frame=train) >>> (fprs, tprs) = gbm.roc(train=True, valid=False, xval=False) >>> tprs
-
-
class
h2o.model.metrics_base.
H2OClusteringModelMetrics
(metric_json, on=None, algo='')[source]¶ Bases:
h2o.model.metrics_base.MetricsBase
-
betweenss
()[source]¶ The Between Cluster Sum-of-Square Error, or None if not present.
- Examples
>>> from h2o.estimators.kmeans import H2OKMeansEstimator >>> iris = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/iris/iris_train.csv") >>> km = H2OKMeansEstimator(k=3, nfolds=3) >>> km.train(x=list(range(4)), training_frame=iris) >>> km.betweenss()
-
tot_withinss
()[source]¶ The Total Within Cluster Sum-of-Square Error, or None if not present.
- Examples
>>> from h2o.estimators.kmeans import H2OKMeansEstimator >>> iris = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/iris/iris_train.csv") >>> km = H2OKMeansEstimator(k=3, nfolds=3) >>> km.train(x=list(range(4)), training_frame=iris) >>> km.tot_withinss()
-
totss
()[source]¶ The Total Sum-of-Square Error to Grand Mean, or None if not present.
- Examples
>>> from h2o.estimators.kmeans import H2OKMeansEstimator >>> iris = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/iris/iris_train.csv") >>> km = H2OKMeansEstimator(k=3, nfolds=3) >>> km.train(x=list(range(4)), training_frame=iris) >>> km.totss()
-
-
class
h2o.model.metrics_base.
H2OCoxPHModelMetrics
(metric_json, on=None, algo='')[source]¶ Bases:
h2o.model.metrics_base.MetricsBase
- Examples
>>> from h2o.estimators.coxph import H2OCoxProportionalHazardsEstimator >>> heart = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/coxph_test/heart.csv") >>> coxph = H2OCoxProportionalHazardsEstimator(start_column="start", ... stop_column="stop", ... ties="breslow") >>> coxph.train(x="age", y="event", training_frame=heart) >>> coxph
-
class
h2o.model.metrics_base.
H2ODimReductionModelMetrics
(metric_json, on=None, algo='')[source]¶
-
class
h2o.model.metrics_base.
H2OHGLMModelMetrics
(metric_json, on=None, algo='HGLM Gaussian Gaussian')[source]¶
-
class
h2o.model.metrics_base.
H2OMultinomialModelMetrics
(metric_json, on=None, algo='')[source]¶ Bases:
h2o.model.metrics_base.MetricsBase
-
confusion_matrix
()[source]¶ Returns a confusion matrix based of H2O’s default prediction threshold for a dataset.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["cylinders"] = cars["cylinders"].asfactor() >>> train, valid = cars.split_frame(ratios=[.8], seed=1234) >>> response_col = "cylinders" >>> distribution = "multinomial" >>> predictors = ["displacement","power","weight","acceleration","year"] >>> gbm = H2OGradientBoostingEstimator(nfolds=3, ... distribution = distribution) >>> gbm.train(x=predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> gbm.confusion_matrix(train)
-
hit_ratio_table
()[source]¶ Retrieve the Hit Ratios.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["cylinders"] = cars["cylinders"].asfactor() >>> train, valid = cars.split_frame(ratios=[.8], seed=1234) >>> response_col = "cylinders" >>> distribution = "multinomial" >>> predictors = ["displacement","power","weight","acceleration","year"] >>> gbm = H2OGradientBoostingEstimator(nfolds=3, ... distribution = distribution) >>> gbm.train(x=predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> gbm.hit_ratio_table()
-
-
class
h2o.model.metrics_base.
H2OOrdinalModelMetrics
(metric_json, on=None, algo='')[source]¶ Bases:
h2o.model.metrics_base.MetricsBase
-
confusion_matrix
()[source]¶ Returns a confusion matrix based of H2O’s default prediction threshold for a dataset.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["cylinders"] = cars["cylinders"].asfactor() >>> train, valid = cars.split_frame(ratios=[.8], seed=1234) >>> response_col = "cylinders" >>> distribution = "multinomial" >>> predictors = ["displacement","power","weight","acceleration","year"] >>> gbm = H2OGradientBoostingEstimator(nfolds=3, ... distribution = distribution) >>> gbm.train(x=predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> gbm.confusion_matrix(train)
-
hit_ratio_table
()[source]¶ Retrieve the Hit Ratios.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["cylinders"] = cars["cylinders"].asfactor() >>> train, valid = cars.split_frame(ratios=[.8], seed=1234) >>> response_col = "cylinders" >>> distribution = "multinomial" >>> predictors = ["displacement","power","weight","acceleration","year"] >>> gbm = H2OGradientBoostingEstimator(nfolds=3, ... distribution = distribution) >>> gbm.train(x=predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> gbm.hit_ratio_table()
-
-
class
h2o.model.metrics_base.
H2ORegressionModelMetrics
(metric_json, on=None, algo='')[source]¶ Bases:
h2o.model.metrics_base.MetricsBase
This class provides an API for inspecting the metrics returned by a regression model.
It is possible to retrieve the R^2 (1 - MSE/variance) and MSE.
- Examples
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "cylinders" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_glm = H2OGeneralizedLinearEstimator() >>> cars_glm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_glm.mse()
-
class
h2o.model.metrics_base.
MetricsBase
(*args, **kwargs)[source]¶ Bases:
h2o.model.metrics_base.MetricsBase
A parent class to house common metrics available for the various Metrics types.
The methods here are available across different model categories.
-
aic
()[source]¶ The AIC for this set of metrics.
- Examples
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator >>> prostate = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/prostate/prostate.csv.zip") >>> prostate[2] = prostate[2].asfactor() >>> prostate[4] = prostate[4].asfactor() >>> prostate[5] = prostate[5].asfactor() >>> prostate[8] = prostate[8].asfactor() >>> predictors = ["AGE","RACE","DPROS","DCAPS","PSA","VOL","GLEASON"] >>> response = "CAPSULE" >>> train, valid = prostate.split_frame(ratios=[.8],seed=1234) >>> pros_glm = H2OGeneralizedLinearEstimator(family="binomial") >>> pros_glm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> pros_glm.aic()
-
auc
()[source]¶ The AUC for this set of metrics.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.auc()
-
aucpr
()[source]¶ The area under the precision recall curve.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.aucpr()
-
gini
()[source]¶ Gini coefficient.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.gini()
-
logloss
()[source]¶ Log loss.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.logloss()
-
mae
()[source]¶ The MAE for this set of metrics.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "cylinders" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(distribution = "poisson", ... seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.mae()
-
classmethod
make
(kvs)[source]¶ Factory method to instantiate a MetricsBase object from the list of key-value pairs.
-
mean_per_class_error
()[source]¶ The mean per class error.
- Examples
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator >>> prostate = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/prostate/prostate.csv.zip") >>> prostate[2] = prostate[2].asfactor() >>> prostate[4] = prostate[4].asfactor() >>> prostate[5] = prostate[5].asfactor() >>> prostate[8] = prostate[8].asfactor() >>> predictors = ["AGE","RACE","DPROS","DCAPS","PSA","VOL","GLEASON"] >>> response = "CAPSULE" >>> train, valid = prostate.split_frame(ratios=[.8],seed=1234) >>> pros_glm = H2OGeneralizedLinearEstimator(family="binomial") >>> pros_glm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> pros_glm.mean_per_class_error()
-
mean_residual_deviance
()[source]¶ The mean residual deviance for this set of metrics.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> airlines= h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/airlines/AirlinesTest.csv.zip") >>> air_gbm = H2OGradientBoostingEstimator() >>> air_gbm.train(x=list(range(9)), ... y=9, ... training_frame=airlines, ... validation_frame=airlines) >>> air_gbm.mean_residual_deviance(train=True,valid=False,xval=False)
-
mse
()[source]¶ The MSE for this set of metrics.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.mse()
-
nobs
()[source]¶ The number of observations.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> perf = cars_gbm.model_performance() >>> perf.nobs()
-
null_degrees_of_freedom
()[source]¶ The null DoF if the model has residual deviance, otherwise None.
- Examples
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator >>> prostate = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/prostate/prostate.csv.zip") >>> prostate[2] = prostate[2].asfactor() >>> prostate[4] = prostate[4].asfactor() >>> prostate[5] = prostate[5].asfactor() >>> prostate[8] = prostate[8].asfactor() >>> predictors = ["AGE","RACE","DPROS","DCAPS","PSA","VOL","GLEASON"] >>> response = "CAPSULE" >>> train, valid = prostate.split_frame(ratios=[.8],seed=1234) >>> pros_glm = H2OGeneralizedLinearEstimator(family="binomial") >>> pros_glm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> pros_glm.null_degrees_of_freedom()
-
null_deviance
()[source]¶ The null deviance if the model has residual deviance, otherwise None.
- Examples
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator >>> prostate = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/prostate/prostate.csv.zip") >>> prostate[2] = prostate[2].asfactor() >>> prostate[4] = prostate[4].asfactor() >>> prostate[5] = prostate[5].asfactor() >>> prostate[8] = prostate[8].asfactor() >>> predictors = ["AGE","RACE","DPROS","DCAPS","PSA","VOL","GLEASON"] >>> response = "CAPSULE" >>> train, valid = prostate.split_frame(ratios=[.8],seed=1234) >>> pros_glm = H2OGeneralizedLinearEstimator(family="binomial") >>> pros_glm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> pros_glm.null_deviance()
-
r2
()[source]¶ The R squared coefficient.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.r2()
-
residual_degrees_of_freedom
()[source]¶ The residual DoF if the model has residual deviance, otherwise None.
- Examples
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator >>> prostate = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/prostate/prostate.csv.zip") >>> prostate[2] = prostate[2].asfactor() >>> prostate[4] = prostate[4].asfactor() >>> prostate[5] = prostate[5].asfactor() >>> prostate[8] = prostate[8].asfactor() >>> predictors = ["AGE","RACE","DPROS","DCAPS","PSA","VOL","GLEASON"] >>> response = "CAPSULE" >>> train, valid = prostate.split_frame(ratios=[.8],seed=1234) >>> pros_glm = H2OGeneralizedLinearEstimator(family="binomial") >>> pros_glm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> pros_glm.residual_degrees_of_freedom()
-
residual_deviance
()[source]¶ The residual deviance if the model has it, otherwise None.
- Examples
>>> from h2o.estimators.glm import H2OGeneralizedLinearEstimator >>> prostate = h2o.import_file("http://s3.amazonaws.com/h2o-public-test-data/smalldata/prostate/prostate.csv.zip") >>> prostate[2] = prostate[2].asfactor() >>> prostate[4] = prostate[4].asfactor() >>> prostate[5] = prostate[5].asfactor() >>> prostate[8] = prostate[8].asfactor() >>> predictors = ["AGE","RACE","DPROS","DCAPS","PSA","VOL","GLEASON"] >>> response = "CAPSULE" >>> train, valid = prostate.split_frame(ratios=[.8],seed=1234) >>> pros_glm = H2OGeneralizedLinearEstimator(family="binomial") >>> pros_glm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> pros_glm.residual_deviance()
-
rmse
()[source]¶ The RMSE for this set of metrics.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.rmse()
-
rmsle
()[source]¶ The RMSLE for this set of metrics.
- Examples
>>> from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "cylinders" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(distribution = "poisson", ... seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.rmsle()
-
show
()[source]¶ Display a short summary of the metrics.
- Examples
>>> from from h2o.estimators.gbm import H2OGradientBoostingEstimator >>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv") >>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor() >>> predictors = ["displacement","power","weight","acceleration","year"] >>> response = "economy_20mpg" >>> train, valid = cars.split_frame(ratios = [.8], seed = 1234) >>> cars_gbm = H2OGradientBoostingEstimator(seed = 1234) >>> cars_gbm.train(x = predictors, ... y = response, ... training_frame = train, ... validation_frame = valid) >>> cars_gbm.show()
-
H2O Grid Metrics
¶
-
class
h2o.grid.metrics.
H2OAutoEncoderGridSearch
[source]¶ Bases:
object
-
anomaly
(test_data, per_feature=False)[source]¶ Obtain the reconstruction error for the input test_data.
- Parameters
test_data (H2OFrame) – The dataset upon which the reconstruction error is computed.
per_feature (bool) – Whether to return the square reconstruction error per feature. Otherwise, return the mean square error.
- Returns
the reconstruction error.
-
-
class
h2o.grid.metrics.
H2OBinomialGridSearch
[source]¶ Bases:
object
-
F0point5
(thresholds=None, train=False, valid=False, xval=False)[source]¶ Get the F0.5 for a set of thresholds.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
train (bool) – If train is True, then return the F0point5 value for the training data.
valid (bool) – If valid is True, then return the F0point5 value for the validation data.
xval (bool) – If xval is True, then return the F0point5 value for the cross validation data.
- Returns
The F0point5 for this binomial model.
-
F1
(thresholds=None, train=False, valid=False, xval=False)[source]¶ Get the F1 values for a set of thresholds for the models explored.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
train (bool) – If True, return the F1 value for the training data.
valid (bool) – If True, return the F1 value for the validation data.
xval (bool) – If True, return the F1 value for each of the cross-validated splits.
- Returns
Dictionary of model keys to F1 values
-
F2
(thresholds=None, train=False, valid=False, xval=False)[source]¶ Get the F2 for a set of thresholds.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
train (bool) – If train is True, then return the F2 value for the training data.
valid (bool) – If valid is True, then return the F2 value for the validation data.
xval (bool) – If xval is True, then return the F2 value for the cross validation data.
- Returns
Dictionary of model keys to F2 values.
-
accuracy
(thresholds=None, train=False, valid=False, xval=False)[source]¶ Get the accuracy for a set of thresholds.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
train (bool) – If train is True, then return the accuracy value for the training data.
valid (bool) – If valid is True, then return the accuracy value for the validation data.
xval (bool) – If xval is True, then return the accuracy value for the cross validation data.
- Returns
The accuracy for this binomial model.
-
confusion_matrix
(metrics=None, thresholds=None, train=False, valid=False, xval=False)[source]¶ Get the confusion matrix for the specified metrics/thresholds.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
metrics – A string (or list of strings) among metrics listed in
H2OBinomialModelMetrics.maximizing_metrics
. Defaults to ‘f1’.thresholds – A value (or list of values) between 0 and 1. If None, then the thresholds maximizing each provided metric will be used.
train (bool) – If train is True, then return the confusion matrix value for the training data.
valid (bool) – If valid is True, then return the confusion matrix value for the validation data.
xval (bool) – If xval is True, then return the confusion matrix value for the cross validation data.
- Returns
The confusion matrix for this binomial model.
-
error
(thresholds=None, train=False, valid=False, xval=False)[source]¶ Get the error for a set of thresholds.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold minimizing the error will be used.
train (bool) – If train is True, then return the error value for the training data.
valid (bool) – If valid is True, then return the error value for the validation data.
xval (bool) – If xval is True, then return the error value for the cross validation data.
- Returns
The error for this binomial model.
-
fallout
(thresholds=None, train=False, valid=False, xval=False)[source]¶ Get the Fallout (AKA False Positive Rate) for a set of thresholds.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
train (bool) – If train is True, then return the fallout value for the training data.
valid (bool) – If valid is True, then return the fallout value for the validation data.
xval (bool) – If xval is True, then return the fallout value for the cross validation data.
- Returns
The fallout for this binomial model.
-
find_idx_by_threshold
(threshold, train=False, valid=False, xval=False)[source]¶ Retrieve the index in this metric’s threshold list at which the given threshold is located.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
threshold (float) – The threshold value to search for.
train (bool) – If train is True, then return the idx_by_threshold for the training data.
valid (bool) – If valid is True, then return the idx_by_threshold for the validation data.
xval (bool) – If xval is True, then return the idx_by_threshold for the cross validation data.
- Returns
The idx_by_threshold for this binomial model.
-
find_threshold_by_max_metric
(metric, train=False, valid=False, xval=False)[source]¶ If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
metric (str) – A metric among the metrics listed in
H2OBinomialModelMetrics.maximizing_metrics
.train (bool) – If train is True, then return the threshold_by_max_metric value for the training data.
valid (bool) – If valid is True, then return the threshold_by_max_metric value for the validation data.
xval (bool) – If xval is True, then return the threshold_by_max_metric value for the cross validation data.
- Returns
The threshold_by_max_metric for this binomial model.
-
fnr
(thresholds=None, train=False, valid=False, xval=False)[source]¶ Get the False Negative Rates for a set of thresholds. If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
train (bool) – If train is True, then return the FNR value for the training data.
valid (bool) – If valid is True, then return the FNR value for the validation data.
xval (bool) – If xval is True, then return the FNR value for the cross validation data.
- Returns
The FNR for this binomial model.
-
fpr
(thresholds=None, train=False, valid=False, xval=False)[source]¶ Get the False Positive Rates for a set of thresholds.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
train (bool) – If train is True, then return the FPR value for the training data.
valid (bool) – If valid is True, then return the FPR value for the validation data.
xval (bool) – If xval is True, then return the FPR value for the cross validation data.
- Returns
The FPR for this binomial model.
-
max_per_class_error
(thresholds=None, train=False, valid=False, xval=False)[source]¶ Get the max per class error for a set of thresholds.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold minimizing the error will be used.
train (bool) – If train is True, then return the max_per_class_error value for the training data.
valid (bool) – If valid is True, then return the max_per_class_error value for the validation data.
xval (bool) – If xval is True, then return the max_per_class_error value for the cross validation data.
- Returns
The max per class error for this binomial model.
-
mcc
(thresholds=None, train=False, valid=False, xval=False)[source]¶ Get the MCC for a set of thresholds.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
train (bool) – If train is True, then return the mcc value for the training data.
valid (bool) – If valid is True, then return the mcc value for the validation data.
xval (bool) – If xval is True, then return the mcc value for the cross validation data.
- Returns
The MCC for this binomial model.
-
mean_per_class_error
(thresholds=None, train=False, valid=False, xval=False)[source]¶ Get the mean per class error for a set of thresholds.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold minimizing the error will be used.
train (bool) – If train is True, then return the mean_per_class_error value for the training data.
valid (bool) – If valid is True, then return the mean_per_class_error value for the validation data.
xval (bool) – If xval is True, then return the mean_per_class_error value for the cross validation data.
- Returns
The mean per class error for this binomial model.
-
metric
(metric, thresholds=None, train=False, valid=False, xval=False)[source]¶ Get the metric value for a set of thresholds.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
metric – name of the metric to compute.
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
train (bool) – If train is True, then return the metrics for the training data.
valid (bool) – If valid is True, then return the metrics for the validation data.
xval (bool) – If xval is True, then return the metrics for the cross validation data.
- Returns
The metrics for this binomial model.
-
missrate
(thresholds=None, train=False, valid=False, xval=False)[source]¶ Get the miss rate (AKA False Negative Rate) for a set of thresholds.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
train (bool) – If train is True, then return the missrate value for the training data.
valid (bool) – If valid is True, then return the missrate value for the validation data.
xval (bool) – If xval is True, then return the missrate value for the cross validation data.
- Returns
The missrate for this binomial model.
-
precision
(thresholds=None, train=False, valid=False, xval=False)[source]¶ Get the precision for a set of thresholds.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
train (bool) – If train is True, then return the precision value for the training data.
valid (bool) – If valid is True, then return the precision value for the validation data.
xval (bool) – If xval is True, then return the precision value for the cross validation data.
- Returns
The precision for this binomial model.
-
recall
(thresholds=None, train=False, valid=False, xval=False)[source]¶ Get the Recall (AKA True Positive Rate) for a set of thresholds.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
train (bool) – If train is True, then return the recall value for the training data.
valid (bool) – If valid is True, then return the recall value for the validation data.
xval (bool) – If xval is True, then return the recall value for the cross validation data.
- Returns
The recall for this binomial model.
-
roc
(train=False, valid=False, xval=False)[source]¶ Return the coordinates of the ROC curve for a given set of data, as a two-tuple containing the false positive rates as a list and true positive rates as a list.
If all are False (default), then return the training data. If more than one ROC curve is requested, the data is returned as a dictionary of two-tuples.
- Parameters
train (bool) – If train is true, then return the ROC coordinates for the training data.
valid (bool) – If valid is true, then return the ROC coordinates for the validation data.
xval (bool) – If xval is true, then return the ROC coordinates for the cross validation data.
- Returns
the true cooridinates of the roc curve.
-
sensitivity
(thresholds=None, train=False, valid=False, xval=False)[source]¶ Get the sensitivity (AKA True Positive Rate or Recall) for a set of thresholds.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
train (bool) – If train is True, then return the sensitivity value for the training data.
valid (bool) – If valid is True, then return the sensitivity value for the validation data.
xval (bool) – If xval is True, then return the sensitivity value for the cross validation data.
- Returns
The sensitivity for this binomial model.
-
specificity
(thresholds=None, train=False, valid=False, xval=False)[source]¶ Get the specificity (AKA True Negative Rate) for a set of thresholds.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
train (bool) – If train is True, then return the specificity value for the training data.
valid (bool) – If valid is True, then return the specificity value for the validation data.
xval (bool) – If xval is True, then return the specificity value for the cross validation data.
- Returns
The specificity for this binomial model.
-
tnr
(thresholds=None, train=False, valid=False, xval=False)[source]¶ Get the True Negative Rate for a set of thresholds.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
train (bool) – If train is True, then return the TNR value for the training data.
valid (bool) – If valid is True, then return the TNR value for the validation data.
xval (bool) – If xval is True, then return the TNR value for the cross validation data.
- Returns
The TNR for this binomial model.
-
tpr
(thresholds=None, train=False, valid=False, xval=False)[source]¶ Get the True Positive Rate for a set of thresholds.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the threshold maximizing the metric will be used.
train (bool) – If train is True, then return the TPR value for the training data.
valid (bool) – If valid is True, then return the TPR value for the validation data.
xval (bool) – If xval is True, then return the TPR value for the cross validation data.
- Returns
The TPR for this binomial model.
-
-
class
h2o.grid.metrics.
H2OClusteringGridSearch
[source]¶ Bases:
object
-
betweenss
(train=False, valid=False, xval=False)[source]¶ Get the between cluster sum of squares.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
train (bool) – If True, then return the between cluster sum of squares value for the training data.
valid (bool) – If True, then return the between cluster sum of squares value for the validation data.
xval (bool) – If True, then return the between cluster sum of squares value for each of the cross-validated splits.
- Returns
the between cluster sum of squares values for the specified key(s).
-
centroid_stats
(train=False, valid=False, xval=False)[source]¶ Get the centroid statistics for each cluster.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
train (bool) – If True, then return the centroid statistics for the training data.
valid (bool) – If True, then return the centroid statistics for the validation data.
xval (bool) – If True, then return the centroid statistics for each of the cross-validated splits.
- Returns
the centroid statistics for the specified key(s).
-
num_iterations
()[source]¶ Get the number of iterations that it took to converge or reach max iterations.
-
size
(train=False, valid=False, xval=False)[source]¶ Get the sizes of each cluster.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
train (bool) – If True, then return the cluster sizes for the training data.
valid (bool) – If True, then return the cluster sizes for the validation data.
xval (bool) – If True, then return the cluster sizes for each of the cross-validated splits.
- Returns
the cluster sizes for the specified key(s).
-
tot_withinss
(train=False, valid=False, xval=False)[source]¶ Get the total within cluster sum of squares.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
train (bool) – If True, then return the total within cluster sum of squares for the training data.
valid (bool) – If True, then return the total within cluster sum of squares for the validation data.
xval (bool) – If True, then return the total within cluster sum of squares for each of the cross-validated splits.
- Returns
the total within cluster sum of squares values for the specified key(s).
-
totss
(train=False, valid=False, xval=False)[source]¶ Get the total sum of squares.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
train (bool) – If True, then return total sum of squares for the training data.
valid (bool) – If True, then return the total sum of squares for the validation data.
xval (bool) – If True, then return the total sum of squares for each of the cross-validated splits.
- Returns
the total sum of squares values for the specified key(s).
-
withinss
(train=False, valid=False, xval=False)[source]¶ Get the within cluster sum of squares for each cluster.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
train (bool) – If True, then return within cluster sum of squares for the training data.
valid (bool) – If True, then return the within cluster sum of squares for the validation data.
xval (bool) – If True, then return the within cluster sum of squares for each of the cross-validated splits.
- Returns
the within cluster sum of squares values for the specified key(s).
-
-
class
h2o.grid.metrics.
H2ODimReductionGridSearch
[source]¶ Bases:
object
-
class
h2o.grid.metrics.
H2OMultinomialGridSearch
[source]¶ Bases:
object
-
confusion_matrix
(data)[source]¶ Returns a confusion matrix based of H2O’s default prediction threshold for a dataset.
- Parameters
data – metric for which the confusion matrix will be calculated.
-
hit_ratio_table
(train=False, valid=False, xval=False)[source]¶ Retrieve the Hit Ratios.
If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
train (bool) – If train is True, then return the hit ratio value for the training data.
valid (bool) – If valid is True, then return the hit ratio value for the validation data.
xval (bool) – If xval is True, then return the hit ratio value for the cross validation data.
- Returns
The hit ratio for this multinomial model.
-
mean_per_class_error
(train=False, valid=False, xval=False)[source]¶ Get the mean per class error.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
train (bool) – If train is True, then return the mean per class error value for the training data.
valid (bool) – If valid is True, then return the mean per class error value for the validation data.
xval (bool) – If xval is True, then return the mean per class error value for the cross validation data.
- Returns
The mean per class error for this multinomial model.
-
-
class
h2o.grid.metrics.
H2OOrdinalGridSearch
[source]¶ Bases:
object
-
confusion_matrix
(data)[source]¶ Returns a confusion matrix based of H2O’s default prediction threshold for a dataset.
- Parameters
data – metric for which the confusion matrix will be calculated.
-
hit_ratio_table
(train=False, valid=False, xval=False)[source]¶ Retrieve the Hit Ratios.
If all are False (default), then return the training metric value. If more than one option is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
train (bool) – If train is True, then return the hit ratio value for the training data.
valid (bool) – If valid is True, then return the hit ratio value for the validation data.
xval (bool) – If xval is True, then return the hit ratio value for the cross validation data.
- Returns
The hit ratio for this ordinal model.
-
mean_per_class_error
(train=False, valid=False, xval=False)[source]¶ Get the mean per class error.
If all are False (default), then return the training metric value. If more than one options is set to True, then return a dictionary of metrics where the keys are “train”, “valid”, and “xval”.
- Parameters
train (bool) – If train is True, then return the mean per class error value for the training data.
valid (bool) – If valid is True, then return the mean per class error value for the validation data.
xval (bool) – If xval is True, then return the mean per class error value for the cross validation data.
- Returns
The mean per class error for this ordinal model.
-