nestedtune

You tune a model with cross-validation and keep the setting with the best score. That score is optimistic, because you picked the winner for scoring well. nestedtune gives you an honest number instead. It builds a nested resampling design. It tunes each outer fold on its own inner resamples with tune or finetune. It selects the fold’s winner, fits it, and scores that fit on rows the tuning never saw. It keeps what every fold chose.

Those steps together, resample, tune, select, fit, are the procedure. The mean of the outer scores estimates how well that procedure performs on new data. The model to deploy is fitted afterwards by the same procedure on all the data. It is a separate object with no performance number of its own.

Installation

# install.packages("pak")
pak::pak("tidymodels/nestedtune")

Example

library(tidymodels)
library(nestedtune)

set.seed(1)
folds <- nested_resamples(
  mtcars,
  outside = vfold_cv(v = 5),
  inside = vfold_cv(v = 5)
)

wf <- workflow(
  mpg ~ .,
  rand_forest(mtry = tune(), min_n = tune()) |>
    set_engine("ranger") |>
    set_mode("regression")
)
grid <- expand.grid(mtry = c(2L, 5L, 8L), min_n = c(2L, 10L))

set.seed(2)
res <- nested_tune_grid(wf, folds, grid = grid)

# The estimate for the procedure. Report this.
collect_metrics(res)
#> # A tibble: 2 × 5
#>   .metric .estimator  mean     n std_err
#>   <chr>   <chr>      <dbl> <int>   <dbl>
#> 1 rmse    standard   2.46      5  0.445 
#> 2 rsq     standard   0.844     5  0.0267

# The model to deploy, fitted by the same procedure on all the data.
set.seed(3)
final <- nested_final_fit(wf, res)
predict(final, new_data = mtcars[1:3, ])
#> # A tibble: 3 × 1
#>   .pred
#>   <dbl>
#> 1  20.9
#> 2  20.9
#> 3  23.8

Learn more:

• Nested cross-validation, the path from a design to a write-up.
• What the estimate means, which quantity the nested number is and what it is not.
• Choosing the inner tuner, the Bayesian search, the two racing searches, simulated annealing and a set of workflows on one design.
• Reading the results, every column of the results object and every function that reads it.
• Running the outer loop in parallel, the same call on a pool of mirai daemons.