max_depth¶
- Available in: GBM, DRF, XGBoost, Isolation Forest
- Hyperparameter: yes
Description¶
This specifies the maximum depth to which each tree will be built. A single tree will stop splitting when there are no more splits that satisfy the min_rows parameter, if it reaches max_depth, or if there are no splits that satisfy this min_split_improvement parameter.
In general, deeper trees can seem to provide better accuracy on a training set because deeper trees can overfit your model to your data. Also, the deeper the algorithm goes, the more computing time is required. This is especially true at depths greater than 10. At depth 4, 8 nodes, for example, you need 8 * 100 * 20 trials to complete this splitting for the layer.
One way to determine an appropriate value for max_depth is to run a quick Cartesian grid search. Each model in the grid search will use early stopping to tune the number of trees using the validation set AUC, as before. The examples below are also available in the GBM Tuning Tutorials folder on GitHub.
The max_depth default value varies depending on the algorithm.
- GBM: default is 5.
- DRF: default is 20.
- XGBoost: default is 6.
- Isolation Forest: default is 8.
Example¶
library(h2o)
h2o.init()
# import the titanic dataset
df <- h2o.importFile(path = "http://s3.amazonaws.com/h2o-public-test-data/smalldata/gbm_test/titanic.csv")
dim(df)
head(df)
tail(df)
summary(df,exact_quantiles=TRUE)
# pick a response for the supervised problem
response <- "survived"
# the response variable is an integer.
# we will turn it into a categorical/factor for binary classification
df[[response]] <- as.factor(df[[response]])
# use all other columns (except for the name) as predictors
predictors <- setdiff(names(df), c(response, "name"))
# split the data for machine learning
splits <- h2o.splitFrame(data = df,
ratios = c(0.6,0.2),
destination_frames = c("train.hex", "valid.hex", "test.hex"),
seed = 1234)
train <- splits[[1]]
valid <- splits[[2]]
test <- splits[[3]]
# Establish a baseline performance using a default GBM model trained on the 60% training split
# We only provide the required parameters, everything else is default
gbm <- h2o.gbm(x = predictors, y = response, training_frame = train)
# Get the AUC on the validation set
h2o.auc(h2o.performance(gbm, newdata = valid))
# The AUC is over 94%, so this model is highly predictive!
[1] 0.9480135
# Determine the best max_depth value to use during a hyper-parameter search.
# Depth 10 is usually plenty of depth for most datasets, but you never know
hyper_params = list( max_depth = seq(1,29,2) )
# or hyper_params = list( max_depth = c(4,6,8,12,16,20) ), which is faster for larger datasets
grid <- h2o.grid(
hyper_params = hyper_params,
# full Cartesian hyper-parameter search
search_criteria = list(strategy = "Cartesian"),
# which algorithm to run
algorithm="gbm",
# identifier for the grid, to later retrieve it
grid_id="depth_grid",
# standard model parameters
x = predictors,
y = response,
training_frame = train,
validation_frame = valid,
# more trees is better if the learning rate is small enough
# here, use "more than enough" trees - we have early stopping
ntrees = 10000,
# smaller learning rate is better, but because we have learning_rate_annealing,
# we can afford to start with a bigger learning rate
learn_rate = 0.05,
# learning rate annealing: learning_rate shrinks by 1% after every tree
# (use 1.00 to disable, but then lower the learning_rate)
learn_rate_annealing = 0.99,
# sample 80% of rows per tree
sample_rate = 0.8,
# sample 80% of columns per split
col_sample_rate = 0.8,
# fix a random number generator seed for reproducibility
seed = 1234,
# early stopping once the validation AUC doesn't improve by at least
# 0.01% for 5 consecutive scoring events
stopping_rounds = 5,
stopping_tolerance = 1e-4,
stopping_metric = "AUC",
# score every 10 trees to make early stopping reproducible
# (it depends on the scoring interval)
score_tree_interval = 10)
# by default, display the grid search results sorted by increasing logloss
# (because this is a classification task)
grid
# sort the grid models by decreasing AUC
sortedGrid <- h2o.getGrid("depth_grid", sort_by="auc", decreasing = TRUE)
sortedGrid
# find the range of max_depth for the top 5 models
topDepths = sortedGrid@summary_table$max_depth[1:5]
minDepth = min(as.numeric(topDepths))
maxDepth = max(as.numeric(topDepths))
> sortedGrid
#H2O Grid Details
Grid ID: depth_grid
Used hyper parameters:
- max_depth
Number of models: 15
Number of failed models: 0
Hyper-Parameter Search Summary: ordered by decreasing auc
max_depth model_ids auc
1 13 depth_grid_model_6 0.9552831783601015
2 27 depth_grid_model_13 0.9547196393350239
3 17 depth_grid_model_8 0.9543251620174698
4 11 depth_grid_model_5 0.9538743307974078
5 9 depth_grid_model_4 0.9534798534798535
6 19 depth_grid_model_9 0.9534234995773457
7 25 depth_grid_model_12 0.9529726683572838
8 29 depth_grid_model_14 0.9528036066497605
9 21 depth_grid_model_10 0.9526908988447449
10 15 depth_grid_model_7 0.9526345449422373
11 7 depth_grid_model_3 0.951789236404621
12 23 depth_grid_model_11 0.9505494505494505
13 3 depth_grid_model_1 0.949084249084249
14 5 depth_grid_model_2 0.9484361792054099
15 1 depth_grid_model_0 0.9478162862778248
import h2o
h2o.init()
from h2o.estimators.gbm import H2OGradientBoostingEstimator
from h2o.grid.grid_search import H2OGridSearch
# import the titanic dataset
df = h2o.import_file(path = "http://s3.amazonaws.com/h2o-public-test-data/smalldata/gbm_test/titanic.csv")
# pick a response for the supervised problem
response = "survived"
# the response variable is an integer
# we will turn it into a categorical/factor for binary classification
df[response] = df[response].asfactor()
# use all other columns as predictors
# (except for the name & the response column ("survived"))
predictors = df.columns
del predictors[1:3]
# split the data for machine learning
train, valid, test = df.split_frame(
ratios=[0.6,0.2],
seed=1234,
destination_frames=['train.hex','valid.hex','test.hex']
)
# Establish baseline performance
# We only provide the required parameters, everything else is default
gbm = H2OGradientBoostingEstimator()
gbm.train(x=predictors, y=response, training_frame=train)
# Get the AUC on the validation set
perf = gbm.model_performance(valid)
print perf.auc()
# The AUC is over 94%, so this model is highly predictive!
0.948013524937
# Determine the best max_depth value to use during a hyper-parameter search
# Depth 10 is usually plenty of depth for most datasets, but you never know
hyper_params = {'max_depth' : range(1,30,2)}
# hyper_params = {max_depth = [4,6,8,12,16,20]} may be faster for larger datasets
#Build initial GBM Model
gbm_grid = H2OGradientBoostingEstimator(
# more trees is better if the learning rate is small enough
# here, use "more than enough" trees - we have early stopping
ntrees=10000,
# smaller learning rate is better
# since we have learning_rate_annealing, we can afford to start with a
# bigger learning rate
learn_rate=0.05,
# learning rate annealing: learning_rate shrinks by 1% after every tree
# (use 1.00 to disable, but then lower the learning_rate)
learn_rate_annealing = 0.99,
# sample 80% of rows per tree
sample_rate = 0.8,
# sample 80% of columns per split
col_sample_rate = 0.8,
# fix a random number generator seed for reproducibility
seed = 1234,
# score every 10 trees to make early stopping reproducible
# (it depends on the scoring interval)
score_tree_interval = 10,
# early stopping once the validation AUC doesn't improve by at least 0.01% for
# 5 consecutive scoring events
stopping_rounds = 5,
stopping_metric = "AUC",
stopping_tolerance = 1e-4)
# Build grid search with previously made GBM and hyper parameters
grid = H2OGridSearch(gbm_grid,hyper_params,
grid_id = 'depth_grid',
search_criteria = {'strategy': "Cartesian"})
# Train grid search
grid.train(x=predictors,
y=response,
training_frame = train,
validation_frame = valid)
# Display the grid search results
# Sorted by increasing logloss (because this is a classification task)
print grid
max_depth model_ids logloss
0 17 depth_grid_model_8 0.20544094075930078
1 19 depth_grid_model_9 0.20584402503242194
2 27 depth_grid_model_13 0.20627418156921704
3 11 depth_grid_model_5 0.2069364255413584
4 13 depth_grid_model_6 0.2078569528636169
5 25 depth_grid_model_12 0.20834760530631993
6 9 depth_grid_model_4 0.20842232867415922
7 29 depth_grid_model_14 0.20904163538087436
8 15 depth_grid_model_7 0.20991531457742935
9 23 depth_grid_model_11 0.2104361858121492
10 21 depth_grid_model_10 0.21069590143686837
11 7 depth_grid_model_3 0.21127939637392396
12 5 depth_grid_model_2 0.21509420086032935
13 3 depth_grid_model_1 0.21854010261642962
14 1 depth_grid_model_0 0.23892331983893703
# Sort the grid models by decreasing AUC
sorted_grid = grid.get_grid(sort_by='auc',decreasing=True)
print(sorted_grid)
max_depth model_ids auc
0 13 depth_grid_model_6 0.9552831783601015
1 27 depth_grid_model_13 0.9547196393350239
2 17 depth_grid_model_8 0.9543251620174698
3 11 depth_grid_model_5 0.9538743307974078
4 9 depth_grid_model_4 0.9534798534798535
5 19 depth_grid_model_9 0.9534234995773457
6 25 depth_grid_model_12 0.9529726683572838
7 29 depth_grid_model_14 0.9528036066497605
8 21 depth_grid_model_10 0.9526908988447449
9 15 depth_grid_model_7 0.9526345449422373
10 7 depth_grid_model_3 0.951789236404621
11 23 depth_grid_model_11 0.9505494505494505
12 3 depth_grid_model_1 0.949084249084249
13 5 depth_grid_model_2 0.9484361792054099
14 1 depth_grid_model_0 0.9478162862778248
It appears that max_depth values of 9 to 27 are best suited for this dataset, which is unusally deep.