fold_assignment

  • Available in: GBM, DRF, Deep Learning, GLM, GAM, Naïve-Bayes, K-Means, XGBoost

  • Hyperparameter: no

Description

This option specifies the scheme to use for cross-validation fold assignment. This option is only applicable if a value for nfolds is specified and a fold_column is not specified. Options include:

  • Auto: Allow the algorithm to automatically choose an option. Auto currently uses Random.

  • Random: Randomly split the data into nfolds pieces. (Default)

  • Modulo: Performs modulo operation when splitting the folds.

  • Stratified: Stratifies the folds based on the response variable for classification problems.

Keep the following in mind when specifying a fold assignment for your data:

  • Random is best for large datasets, but can lead to imbalanced samples for small datasets.

  • Modulo is a simple deterministic way to evenly split the dataset into the folds and does not depend on the seed.

  • Specifying Stratified will attempt to evenly distribute observations from the different classes to all sets when splitting a dataset into training and validation. This can be useful if there are many classes and the dataset is relatively small.

  • Note that all three options are only suitable for datasets that are i.i.d. If the dataset requires custom grouping to perform meaningful cross-validation, then a fold_column should be created and provided instead.

  • In general, when comparing multiple models using validation sets, ensure that you use the same validation set for all models. When performing cross-validation, specify a seed for all models, or specify Modulo for the fold_assignment. This ensures that the cross-validation folds are the same, and eliminates the noise that can come from, for example, the Random fold assignment.

Example

R

Python

library(h2o)
h2o.init()

# import the cars dataset:
# this dataset is used to classify whether or not a car is economical based on
# the car's displacement, power, weight, and acceleration, and the year it was made
cars <- h2o.importFile("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")

# convert response column to a factor
cars["economy_20mpg"] <- as.factor(cars["economy_20mpg"])

# set the predictor names and the response column name
predictors <- c("displacement", "power", "weight", "acceleration", "year")
response <- "economy_20mpg"

# try using the fold_assignment parameter:
# note you must set nfolds to use this parameter
assignment_type <- "Random"
# you can also try "Auto", "Modulo", and "Stratified"

# train a GBM
car_gbm <- h2o.gbm(x = predictors, y = response, training_frame = cars,
                   fold_assignment = assignment_type,
                   nfolds = 5, seed = 1234)

# print the auc for your validation data
print(h2o.auc(car_gbm, xval = TRUE))

import h2o
from h2o.estimators.gbm import H2OGradientBoostingEstimator
h2o.init()

# import the cars dataset:
# this dataset is used to classify whether or not a car is economical based on
# the car's displacement, power, weight, and acceleration, and the year it was made
cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")

# convert response column to a factor
cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()

# set the predictor names and the response column name
predictors = ["displacement","power","weight","acceleration","year"]
response = "economy_20mpg"

# try using the fold_assignment parameter:
# note you must set nfolds to use this parameter
assignment_type = "Random"
# you can also try "Auto", "Modulo", and "Stratified"

# Initialize and train a GBM
cars_gbm = H2OGradientBoostingEstimator(fold_assignment = assignment_type, nfolds = 5, seed = 1234)
cars_gbm.train(x = predictors, y = response, training_frame = cars)

# print the auc for the validation data
cars_gbm.auc(xval=True)