Skip to contents

Calculate a bootstrap confidence interval for the performance on a single train/test split

Source: R/performance.R
bootstrap_performance.Rd

Uses rsample::bootstraps(), rsample::int_pctl(), and furrr::future_map()

Usage

bootstrap_performance(
  ml_result,
  outcome_colname,
  bootstrap_times = 10000,
  alpha = 0.05
)

Arguments

ml_result

result returned from a single run_ml() call

outcome_colname

Column name as a string of the outcome variable (default NULL; the first column will be chosen automatically).

bootstrap_times

the number of boostraps to create (default: 10000)

alpha

the alpha level for the confidence interval (default 0.05 to create a 95% confidence interval)

Value

a data frame with an estimate (.estimate), lower bound (.lower), and upper bound (.upper) for each performance metric (term).

Author

Kelly Sovacool, sovacool@umich.edu

Examples

bootstrap_performance(otu_mini_bin_results_glmnet, "dx",
  bootstrap_times = 10, alpha = 0.10
)
#> Warning: Recommend at least 1000 non-missing bootstrap resamples for terms `AUC`,
#> `Accuracy`, `Balanced_Accuracy`, `Detection_Rate`, `F1`, `Kappa`,
#> `Neg_Pred_Value`, `Pos_Pred_Value`, `Precision`, `Recall`, `Sensitivity`,
#> `Specificity`, `cv_metric_AUC`, `logLoss`, and `prAUC`.
#> # A tibble: 15 × 6
#>    term              .lower .estimate .upper .alpha .method   
#>    <chr>              <dbl>     <dbl>  <dbl>  <dbl> <chr>     
#>  1 AUC                0.476     0.617  0.737    0.1 percentile
#>  2 Accuracy           0.408     0.564  0.695    0.1 percentile
#>  3 Balanced_Accuracy  0.406     0.562  0.688    0.1 percentile
#>  4 Detection_Rate     0.154     0.279  0.390    0.1 percentile
#>  5 F1                 0.363     0.553  0.714    0.1 percentile
#>  6 Kappa             -0.187     0.123  0.376    0.1 percentile
#>  7 Neg_Pred_Value     0.438     0.563  0.744    0.1 percentile
#>  8 Pos_Pred_Value     0.363     0.561  0.697    0.1 percentile
#>  9 Precision          0.363     0.561  0.697    0.1 percentile
#> 10 Recall             0.353     0.552  0.732    0.1 percentile
#> 11 Sensitivity        0.353     0.552  0.732    0.1 percentile
#> 12 Specificity        0.428     0.573  0.700    0.1 percentile
#> 13 cv_metric_AUC      0.622     0.622  0.622    0.1 percentile
#> 14 logLoss            0.672     0.687  0.702    0.1 percentile
#> 15 prAUC              0.457     0.578  0.682    0.1 percentile
if (FALSE) { # \dontrun{
outcome_colname <- "dx"
run_ml(otu_mini_bin, "rf", outcome_colname = "dx") %>%
  bootstrap_performance(outcome_colname,
    bootstrap_times = 10000,
    alpha = 0.05
  )
} # }