Skip to contents

Calculate a bootstrap confidence interval for the performance on a single train/test split

Source: R/performance.R
bootstrap_performance.Rd

Uses rsample::bootstraps(), rsample::int_pctl(), and furrr::future_map()

Usage

bootstrap_performance(
  ml_result,
  outcome_colname,
  bootstrap_times = 10000,
  alpha = 0.05
)

Arguments

ml_result

result returned from a single run_ml() call

outcome_colname

Column name as a string of the outcome variable (default NULL; the first column will be chosen automatically).

bootstrap_times

the number of boostraps to create (default: 10000)

alpha

the alpha level for the confidence interval (default 0.05 to create a 95% confidence interval)

Value

a data frame with an estimate (.estimate), lower bound (.lower), and upper bound (.upper) for each performance metric (term).

Author

Kelly Sovacool, sovacool@umich.edu

Examples

bootstrap_performance(otu_mini_bin_results_glmnet, "dx",
  bootstrap_times = 10, alpha = 0.10
)
#> Warning: Recommend at least 1000 non-missing bootstrap resamples for terms: `AUC`, `Accuracy`, `Balanced_Accuracy`, `Detection_Rate`, `F1`, `Kappa`, `Neg_Pred_Value`, `Pos_Pred_Value`, `Precision`, `Recall`, `Sensitivity`, `Specificity`, `cv_metric_AUC`, `logLoss`, `prAUC`.
#> # A tibble: 15 × 6
#>    term               .lower .estimate .upper .alpha .method   
#>    <chr>               <dbl>     <dbl>  <dbl>  <dbl> <chr>     
#>  1 AUC                0.555      0.676  0.779    0.1 percentile
#>  2 Accuracy           0.487      0.595  0.692    0.1 percentile
#>  3 Balanced_Accuracy  0.485      0.594  0.689    0.1 percentile
#>  4 Detection_Rate     0.191      0.279  0.333    0.1 percentile
#>  5 F1                 0.432      0.578  0.659    0.1 percentile
#>  6 Kappa             -0.0313     0.188  0.379    0.1 percentile
#>  7 Neg_Pred_Value     0.456      0.606  0.707    0.1 percentile
#>  8 Pos_Pred_Value     0.383      0.583  0.688    0.1 percentile
#>  9 Precision          0.383      0.583  0.688    0.1 percentile
#> 10 Recall             0.496      0.581  0.649    0.1 percentile
#> 11 Sensitivity        0.496      0.581  0.649    0.1 percentile
#> 12 Specificity        0.402      0.607  0.740    0.1 percentile
#> 13 cv_metric_AUC      0.622      0.622  0.622    0.1 percentile
#> 14 logLoss            0.664      0.678  0.689    0.1 percentile
#> 15 prAUC              0.541      0.628  0.713    0.1 percentile
if (FALSE) {
outcome_colname <- "dx"
run_ml(otu_mini_bin, "rf", outcome_colname = "dx") %>%
  bootstrap_performance(outcome_colname,
    bootstrap_times = 10000,
    alpha = 0.05
  )
}