Metrics in H2O¶
- class h2o.model.metrics_base.H2OAutoEncoderModelMetrics(metric_json, on=None, algo='')[source]¶
Bases: h2o.model.metrics_base.MetricsBase
Methods
aic() return: Retrieve the AIC for this set of metrics. auc() return: Retrieve the AUC for this set of metrics. giniCoef() return: Retrieve the Gini coefficeint for this set of metrics. logloss() return: Retrieve the log loss for this set of metrics. mean_residual_deviance() return: Retrieve the mean residual deviance for this set of metrics. mse() return: Retrieve the MSE for this set of metrics next() null_degrees_of_freedom() return: the null dof if the model has residual deviance, or None if no null dof. null_deviance() return: the null deviance if the model has residual deviance, or None if no null deviance. r2() return: Retrieve the R^2 coefficient for this set of metrics residual_degrees_of_freedom() return: the residual dof if the model has residual deviance, or None if no residual dof. residual_deviance() return: the residual deviance if the model has residual deviance, or None if no residual deviance. show() Display a short summary of the metrics.
- class h2o.model.metrics_base.H2OBinomialModelMetrics(metric_json, on=None, algo='')[source]¶
Bases: h2o.model.metrics_base.MetricsBase
This class is essentially an API for the AUC object. This class contains methods for inspecting the AUC for different criteria. To input the different criteria, use the static variable criteria
Attributes
fprs Return all false positive rates for all threshold values. tprs Return all true positive rates for all threshold values. Methods
F0point5([thresholds]) param thresholds: thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. F1([thresholds]) param thresholds: thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. F2([thresholds]) param thresholds: thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. accuracy([thresholds]) param thresholds: thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. aic() return: Retrieve the AIC for this set of metrics. auc() return: Retrieve the AUC for this set of metrics. confusion_matrix([metrics, thresholds]) Get the confusion matrix for the specified metric error([thresholds]) param thresholds: thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. fallout([thresholds]) param thresholds: thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. find_idx_by_threshold(threshold) Retrieve the index in this metric’s threshold list at which the given threshold is located. find_threshold_by_max_metric(metric) param metric: A string in {“min_per_class_accuracy”, “absolute_MCC”, “precision”, “accuracy”, “f0point5”, “f2”, “f1”} fnr([thresholds]) param thresholds: thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. fpr([thresholds]) param thresholds: thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. gains_lift() Retrieve the Gains/Lift table giniCoef() return: Retrieve the Gini coefficeint for this set of metrics. logloss() return: Retrieve the log loss for this set of metrics. max_per_class_error([thresholds]) param thresholds: thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. mcc([thresholds]) param thresholds: thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. mean_residual_deviance() return: Retrieve the mean residual deviance for this set of metrics. metric(metric[, thresholds]) param metric: The desired metric missrate([thresholds]) param thresholds: thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. mse() return: Retrieve the MSE for this set of metrics next() null_degrees_of_freedom() return: the null dof if the model has residual deviance, or None if no null dof. null_deviance() return: the null deviance if the model has residual deviance, or None if no null deviance. plot([type]) Produce the desired metric plot precision([thresholds]) param thresholds: thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. r2() return: Retrieve the R^2 coefficient for this set of metrics recall([thresholds]) param thresholds: thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. residual_degrees_of_freedom() return: the residual dof if the model has residual deviance, or None if no residual dof. residual_deviance() return: the residual deviance if the model has residual deviance, or None if no residual deviance. sensitivity([thresholds]) param thresholds: thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. show() Display a short summary of the metrics. specificity([thresholds]) param thresholds: thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. tnr([thresholds]) param thresholds: thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. tpr([thresholds]) param thresholds: thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. - F0point5(thresholds=None)[source]¶
Parameters: thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. Returns: The F0point5 for this set of metrics and thresholds.
- F1(thresholds=None)[source]¶
Parameters: thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. Returns: The F1 for the given set of thresholds.
- F2(thresholds=None)[source]¶
Parameters: thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. Returns: The F2 for this set of metrics and thresholds
- accuracy(thresholds=None)[source]¶
Parameters: thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. Returns: The accuracy for this set of metrics and thresholds
- confusion_matrix(metrics=None, thresholds=None)[source]¶
Get the confusion matrix for the specified metric
Parameters: - metrics – A string (or list of strings) in {“min_per_class_accuracy”, “absolute_MCC”, “tnr”, “fnr”, “fpr”, “tpr”, “precision”, “accuracy”, “f0point5”, “f2”, “f1”}
- thresholds – A value (or list of values) between 0 and 1
Returns: a list of ConfusionMatrix objects (if there are more than one to return), or a single ConfusionMatrix (if there is only one)
- error(thresholds=None)[source]¶
Parameters: thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. Returns: The error for this set of metrics and thresholds.
- fallout(thresholds=None)[source]¶
Parameters: thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. Returns: The fallout or False Positive Rate for this set of metrics and thresholds
- find_idx_by_threshold(threshold)[source]¶
Retrieve the index in this metric’s threshold list at which the given threshold is located.
Parameters: threshold – Find the index of this input threshold. Returns: Return the index or throw a ValueError if no such index can be found.
- find_threshold_by_max_metric(metric)[source]¶
Parameters: metric – A string in {“min_per_class_accuracy”, “absolute_MCC”, “precision”, “accuracy”, “f0point5”, “f2”, “f1”} Returns: the threshold at which the given metric is maximum.
- fnr(thresholds=None)[source]¶
Parameters: thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. Returns: The False Negative Rate
- fpr(thresholds=None)[source]¶
Parameters: thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. Returns: The False Positive Rate
- fprs None[source]¶
Return all false positive rates for all threshold values.
Returns: a list of false positive rates.
- max_per_class_error(thresholds=None)[source]¶
Parameters: thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. Returns: Return 1 - min_per_class_accuracy
- mcc(thresholds=None)[source]¶
Parameters: thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. Returns: The absolute MCC (a value between 0 and 1, 0 being totally dissimilar, 1 being identical)
- metric(metric, thresholds=None)[source]¶
Parameters: - metric – The desired metric
- thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used.
Returns: The set of metrics for the list of thresholds
- missrate(thresholds=None)[source]¶
Parameters: thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. Returns: THe missrate or False Negative Rate.
- plot(type='roc', **kwargs)[source]¶
Produce the desired metric plot :param type: the type of metric plot (currently, only ROC supported) :param show: if False, the plot is not shown. matplotlib show method is blocking. :return: None
- precision(thresholds=None)[source]¶
Parameters: thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. Returns: The precision for this set of metrics and thresholds.
- recall(thresholds=None)[source]¶
Parameters: thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. Returns: Recall for this set of metrics and thresholds
- sensitivity(thresholds=None)[source]¶
Parameters: thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. Returns: Sensitivity or True Positive Rate for this set of metrics and thresholds
- specificity(thresholds=None)[source]¶
Parameters: thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. Returns: The specificity or True Negative Rate.
- tnr(thresholds=None)[source]¶
Parameters: thresholds – thresholds parameter must be a list (i.e. [0.01, 0.5, 0.99]). If None, then the thresholds in this set of metrics will be used. Returns: The True Negative Rate
- class h2o.model.metrics_base.H2OClusteringModelMetrics(metric_json, on=None, algo='')[source]¶
Bases: h2o.model.metrics_base.MetricsBase
Methods
aic() return: Retrieve the AIC for this set of metrics. auc() return: Retrieve the AUC for this set of metrics. betweenss() return: the Between Cluster Sum-of-Square Error, or None if not present. giniCoef() return: Retrieve the Gini coefficeint for this set of metrics. logloss() return: Retrieve the log loss for this set of metrics. mean_residual_deviance() return: Retrieve the mean residual deviance for this set of metrics. mse() return: Retrieve the MSE for this set of metrics next() null_degrees_of_freedom() return: the null dof if the model has residual deviance, or None if no null dof. null_deviance() return: the null deviance if the model has residual deviance, or None if no null deviance. r2() return: Retrieve the R^2 coefficient for this set of metrics residual_degrees_of_freedom() return: the residual dof if the model has residual deviance, or None if no residual dof. residual_deviance() return: the residual deviance if the model has residual deviance, or None if no residual deviance. show() Display a short summary of the metrics. tot_withinss() return: the Total Within Cluster Sum-of-Square Error, or None if not present. totss() return: the Total Sum-of-Square Error to Grand Mean, or None if not present.
- class h2o.model.metrics_base.H2ODimReductionModelMetrics(metric_json, on=None, algo='')[source]¶
Bases: h2o.model.metrics_base.MetricsBase
Methods
aic() return: Retrieve the AIC for this set of metrics. auc() return: Retrieve the AUC for this set of metrics. cat_err() return: the Number of Misclassified categories over non-missing categorical entries, or None if not present. giniCoef() return: Retrieve the Gini coefficeint for this set of metrics. logloss() return: Retrieve the log loss for this set of metrics. mean_residual_deviance() return: Retrieve the mean residual deviance for this set of metrics. mse() return: Retrieve the MSE for this set of metrics next() null_degrees_of_freedom() return: the null dof if the model has residual deviance, or None if no null dof. null_deviance() return: the null deviance if the model has residual deviance, or None if no null deviance. num_err() return: the Sum of Squared Error over non-missing numeric entries, or None if not present. r2() return: Retrieve the R^2 coefficient for this set of metrics residual_degrees_of_freedom() return: the residual dof if the model has residual deviance, or None if no residual dof. residual_deviance() return: the residual deviance if the model has residual deviance, or None if no residual deviance. show() Display a short summary of the metrics.
- class h2o.model.metrics_base.H2OMultinomialModelMetrics(metric_json, on=None, algo='')[source]¶
Bases: h2o.model.metrics_base.MetricsBase
Methods
aic() return: Retrieve the AIC for this set of metrics. auc() return: Retrieve the AUC for this set of metrics. confusion_matrix() Returns a confusion matrix based of H2O’s default prediction threshold for a dataset giniCoef() return: Retrieve the Gini coefficeint for this set of metrics. hit_ratio_table() Retrieve the Hit Ratios logloss() return: Retrieve the log loss for this set of metrics. mean_residual_deviance() return: Retrieve the mean residual deviance for this set of metrics. mse() return: Retrieve the MSE for this set of metrics next() null_degrees_of_freedom() return: the null dof if the model has residual deviance, or None if no null dof. null_deviance() return: the null deviance if the model has residual deviance, or None if no null deviance. r2() return: Retrieve the R^2 coefficient for this set of metrics residual_degrees_of_freedom() return: the residual dof if the model has residual deviance, or None if no residual dof. residual_deviance() return: the residual deviance if the model has residual deviance, or None if no residual deviance. show() Display a short summary of the metrics.
- class h2o.model.metrics_base.H2ORegressionModelMetrics(metric_json, on=None, algo='')[source]¶
Bases: h2o.model.metrics_base.MetricsBase
This class provides an API for inspecting the metrics returned by a regression model.
It is possible to retrieve the R^2 (1 - MSE/variance) and MSE
Methods
aic() return: Retrieve the AIC for this set of metrics. auc() return: Retrieve the AUC for this set of metrics. giniCoef() return: Retrieve the Gini coefficeint for this set of metrics. logloss() return: Retrieve the log loss for this set of metrics. mean_residual_deviance() return: Retrieve the mean residual deviance for this set of metrics. mse() return: Retrieve the MSE for this set of metrics next() null_degrees_of_freedom() return: the null dof if the model has residual deviance, or None if no null dof. null_deviance() return: the null deviance if the model has residual deviance, or None if no null deviance. r2() return: Retrieve the R^2 coefficient for this set of metrics residual_degrees_of_freedom() return: the residual dof if the model has residual deviance, or None if no residual dof. residual_deviance() return: the residual deviance if the model has residual deviance, or None if no residual deviance. show() Display a short summary of the metrics.
- class h2o.model.metrics_base.MetricsBase(metric_json, on=None, algo='')[source]¶
Bases: future.types.newobject.newobject
A parent class to house common metrics available for the various Metrics types.
The methods here are available across different model categories, and so appear here.
Methods
aic() return: Retrieve the AIC for this set of metrics. auc() return: Retrieve the AUC for this set of metrics. giniCoef() return: Retrieve the Gini coefficeint for this set of metrics. logloss() return: Retrieve the log loss for this set of metrics. mean_residual_deviance() return: Retrieve the mean residual deviance for this set of metrics. mse() return: Retrieve the MSE for this set of metrics next() null_degrees_of_freedom() return: the null dof if the model has residual deviance, or None if no null dof. null_deviance() return: the null deviance if the model has residual deviance, or None if no null deviance. r2() return: Retrieve the R^2 coefficient for this set of metrics residual_degrees_of_freedom() return: the residual dof if the model has residual deviance, or None if no residual dof. residual_deviance() return: the residual deviance if the model has residual deviance, or None if no residual deviance. show() Display a short summary of the metrics. - mean_residual_deviance()[source]¶
Returns: Retrieve the mean residual deviance for this set of metrics.
- null_degrees_of_freedom()[source]¶
Returns: the null dof if the model has residual deviance, or None if no null dof.
- null_deviance()[source]¶
Returns: the null deviance if the model has residual deviance, or None if no null deviance.
- residual_degrees_of_freedom()[source]¶
Returns: the residual dof if the model has residual deviance, or None if no residual dof.