Tree ensembles and uncertainty

Topics and Objectives

Tree ensembles, bagging, randomization, and bias-variance tradeoffs
Random forests as adaptive weights and data-dependent kernels
Purely random forests, Mondrian partitions, and kernel approximations
AdaBoost, gradient boosting, and the boosting training-error bound
Influence functions, local moment equations, and generalized random forests
Honesty, sample splitting, asymptotic normality, and pointwise confidence intervals
Bootstrap, jackknife, infinitesimal jackknife, and U-statistic variance estimation
Conditional distributions, prediction intervals, and uncertainty in variable importance

Module Schedule

During this three-week module, we will cover the following potential topics as time permits. The presentation schedule for this module will be announced later.

[Bias-Variance Behavior of Kernel Smoothers]
[Review of Tree Model Algorithms]
[Random Forests and Kernel View] [Source Code]
[Adaptiveness of Random Forest Kernels]
[AdaBoost and the Training-Error Bound]
[Gradient Boosting Machines]
[Generalized Random Forests and Influence Functions] [Source Code]
[Variance Estimation in Random Forests] [Source Code]
Conditional distributions and prediction intervals with random forests

Homework

Homework 2 covers both Modules 2 and 3. Please refer to the Module 2 page for the assignment and submission details.

Presentation Session

General logistics and scoring are described on the Present and Challenge page.

Teams assigned to present in this module should select a paper on tree ensembles, random forests, uncertainty quantification for ensemble methods, or another closely related topic. Each team may use up to 12 minutes for its presentation and up to 3-5 minutes for Q&A. The specific time allocation may be shorter depending on the number of teams presenting in the session.

The presentation should focus on one of the following directions:

An algorithm. Explain the problem it addresses, how it is designed, and why it can improve on existing approaches.
A theory. Explain the question it addresses, what the main result tells us, and the setting and assumptions under which it holds.

Numerical implementations or experiments may help illustrate the ideas, but are not required.

Here are some candidate papers. Several study uncertainty, but they do not all target the same quantity. Some are more technically demanding and may require a more focused reading scope. You may also propose a paper of your own choice, but make sure you have a basic understanding of its main ideas and technical demands before committing to it.

Mentch, L., & Zhou, S. (2020). Randomization as Regularization: A Degrees of Freedom Explanation for Random Forest Success. Journal of Machine Learning Research, 21(171), 1-36. [link]
Strobl, C., Boulesteix, A.-L., Zeileis, A., & Hothorn, T. (2007). Bias in Random Forest Variable Importance Measures: Illustrations, Sources and a Solution. BMC Bioinformatics, 8, 25. [link]
Meinshausen, N. (2006). Quantile Regression Forests. Journal of Machine Learning Research, 7(35), 983-999. [link]
Zhang, H., Zimmerman, J., Nettleton, D., & Nordman, D. J. (2020). Random Forest Prediction Intervals. The American Statistician, 74(4), 392-406. [link]
Lu, B., & Hardin, J. (2021). A Unified Framework for Random Forest Prediction Error Estimation. Journal of Machine Learning Research, 22(8), 1-41. [link]
Friedberg, R., Tibshirani, J., Athey, S., & Wager, S. (2021). Local Linear Forests. Journal of Computational and Graphical Statistics, 30(2), 503-517. [link]
Sexton, J., & Laake, P. (2009). Standard Errors for Bagged and Random Forest Estimators. Computational Statistics & Data Analysis, 53(3), 801-811. [link]
Ishwaran, H., & Lu, M. (2019). Standard Errors and Confidence Intervals for Variable Importance in Random Forest Regression, Classification, and Survival. Statistics in Medicine, 38(4), 558-582. [link]
Kim, B., Xu, C., & Barber, R. F. (2020). Predictive Inference Is Free with the Jackknife+-after-Bootstrap. Advances in Neural Information Processing Systems, 33, 4138-4149. [link]
Wager, S., Hastie, T., & Efron, B. (2014). Confidence Intervals for Random Forests: The Jackknife and the Infinitesimal Jackknife. Journal of Machine Learning Research, 15(48), 1625-1651. [link]
Mourtada, J., Gaïffas, S., & Scornet, E. (2021). AMF: Aggregated Mondrian Forests for Online Learning. Journal of the Royal Statistical Society: Series B, 83(3), 505-533. [link]
Ćevid, D., Michel, L., Näf, J., Bühlmann, P., & Meinshausen, N. (2022). Distributional Random Forests: Heterogeneity Adjustment and Multivariate Distributional Regression. Journal of Machine Learning Research, 23(333), 1-79. [link]