Tree ensembles and uncertainty

Topics and Objectives

Tree ensembles, bagging, randomization, and bias-variance tradeoffs
Random forests as adaptive weights and data-dependent kernels
Purely random forests, Mondrian partitions, and kernel approximations
AdaBoost, gradient boosting, and the boosting training-error bound
Influence functions, local moment equations, and generalized random forests
Honesty, sample splitting, asymptotic normality, and pointwise confidence intervals
Bootstrap, jackknife, infinitesimal jackknife, and U-statistic variance estimation
Conditional distributions, prediction intervals, and uncertainty in variable importance

Module Schedule

During this three-week module, we will cover the following potential topics as time permits. The third week will be a Present and Challenge week.

[Bias-Variance Behavior of Kernel Smoothers]
[Review of Tree Model Algorithms]
[Random Forests and Kernel View] [Source Code]
[Adaptiveness of Random Forest Kernels]
[AdaBoost and the Training-Error Bound]
[Gradient Boosting Machines]
[Generalized Random Forests and Influence Functions] [Source Code]
[Variance Estimation in Random Forests] [Source Code]
Conditional distributions and prediction intervals with random forests

Homework

Presentation Session

For this module’s presentation topics, you should select a paper on tree ensembles, random forests, uncertainty quantification for ensemble methods, or another closely related topic.

Several papers below study uncertainty, but they do not all target the same quantity. Teams presenting one of these papers should distinguish prediction intervals for a future response from confidence intervals for a fitted regression target or for variable importance.

Here are some candidate papers. The final four papers are more technically demanding and may require a more focused reading scope:

Mentch, L., & Zhou, S. (2020). Randomization as Regularization: A Degrees of Freedom Explanation for Random Forest Success. Journal of Machine Learning Research, 21(171), 1-36. [link]
Strobl, C., Boulesteix, A.-L., Zeileis, A., & Hothorn, T. (2007). Bias in Random Forest Variable Importance Measures: Illustrations, Sources and a Solution. BMC Bioinformatics, 8, 25. [link]
Meinshausen, N. (2006). Quantile Regression Forests. Journal of Machine Learning Research, 7(35), 983-999. [link]
Zhang, H., Zimmerman, J., Nettleton, D., & Nordman, D. J. (2020). Random Forest Prediction Intervals. The American Statistician, 74(4), 392-406. [link]
Lu, B., & Hardin, J. (2021). A Unified Framework for Random Forest Prediction Error Estimation. Journal of Machine Learning Research, 22(8), 1-41. [link]
Friedberg, R., Tibshirani, J., Athey, S., & Wager, S. (2021). Local Linear Forests. Journal of Computational and Graphical Statistics, 30(2), 503-517. [link]
Sexton, J., & Laake, P. (2009). Standard Errors for Bagged and Random Forest Estimators. Computational Statistics & Data Analysis, 53(3), 801-811. [link]
Ishwaran, H., & Lu, M. (2019). Standard Errors and Confidence Intervals for Variable Importance in Random Forest Regression, Classification, and Survival. Statistics in Medicine, 38(4), 558-582. [link]
Kim, B., Xu, C., & Barber, R. F. (2020). Predictive Inference Is Free with the Jackknife+-after-Bootstrap. Advances in Neural Information Processing Systems, 33, 4138-4149. [link]
Wager, S., Hastie, T., & Efron, B. (2014). Confidence Intervals for Random Forests: The Jackknife and the Infinitesimal Jackknife. Journal of Machine Learning Research, 15(48), 1625-1651. [link]
Mourtada, J., Gaïffas, S., & Scornet, E. (2021). AMF: Aggregated Mondrian Forests for Online Learning. Journal of the Royal Statistical Society: Series B, 83(3), 505-533. [link]
Ćevid, D., Michel, L., Näf, J., Bühlmann, P., & Meinshausen, N. (2022). Distributional Random Forests: Heterogeneity Adjustment and Multivariate Distributional Regression. Journal of Machine Learning Research, 23(333), 1-79. [link]