K-nearest neighbors and nonparametric estimation

Week 5: Sep 23 – Sep 27

Weekly Objectives

• Theoretical Understanding of bias-variance trade-off
• Understand the KNN algorithm
  • The effect of local averaging
  • The effect of \(k\) on the bias-variance trade-off
  • Tune \(k\) using the caret package
  • Various distance measures
  • Computational issue
• Curse of dimensionality
  • high-dimensional issue of local averaging methods
  • Use simulation studies to understand the problem
• Classification
  • Evaluate a classification model

Lecture Notes and R Examples

• K-Nearest Neighbors [Source File]
• The Curse of Dimensionality [Source File]
• Classification Model Evaluation [Source File]

Data

• [Cleveland heart disease data]
• [Handwritten Digit Data] from the ElemStatLearn package

Additional Readings

• Bias-variance Trade-off
• Degrees of Freedom

Homework

• Homework 5 [Source File]
• Due Oct 3, 11:59PM CT
• Solution