Week 5: K-Nearest Neighbors

This week moves from fitted linear models to prediction using nearby observations. K-nearest neighbors estimates a regression function by local averaging and predicts a class by majority voting. We study how the number of neighbors and the choice of distance affect prediction, then examine why finding useful neighbors becomes harder in many dimensions.

Guiding question: When do nearby observations give useful predictions, and what changes when the neighborhood becomes too large or the data have many dimensions?

Learning goals

By the end of this week, you should be able to:

  • explain local averaging for regression and majority voting for classification;
  • describe how kk changes bias and variance, and choose kk by cross-validation;
  • explain how predictor scales and correlations affect distance and neighbor selection;
  • interpret classification error and confusion matrices while distinguishing training performance from test performance; and
  • use the unit-cube argument and repeated simulations to explain the curse of dimensionality and the possible role of lower-dimensional structure.

Lecture notes

  1. K-Nearest Neighbors

    Local averaging, the choice of kk, the bias-variance trade-off, classification, cross-validation, distance measures, and computational issues, with R and Python demonstrations.

  2. The Curse of Dimensionality

    Handwritten digit recognition, neighborhoods in many dimensions, repeated prediction experiments, and the role of lower-dimensional structure.

Homework 05

Homework 05 contains five required problems on KNN bias and variance, comparison with lasso, observed and latent dimension, handwritten digit classification, and a plot-formatting skill. Question 5 asks you to create and install the skill, then explicitly test it on your Question 1 plot. Use either R or Python. All homework assignments are evaluated as Complete or Incomplete. Attempt Questions 1 through 4 before consulting the solutions.

NoteUse of homework submissions

Your Homework 5 submission will be used as example material for peer evaluation and course lectures in future weeks. Use of your material will be restricted to this class and this semester only.

ImportantDeadline

Place homework-05.pdf or homework-05.html in the homework/week-05/ folder of your stat432-fall2026 GitHub repository by Sunday, September 27, 2026, at 11:59 p.m. Central Time. The report must be either a PDF or a self-contained HTML file and must not exceed 10 pages. The final file version shown on GitHub is treated as the submitted version, and its GitHub timestamp determines whether it is on time.

Discussion Session on Tuesday, September 29, 2026

ImportantSubmit Discussion Question

Submit discussion/week-05-question.md by Sunday, September 27, 2026, at 11:59 p.m. Central Time. This required weekly discussion question counts toward the Weekly GitHub Discussion Questions component, worth 10% of the course grade.

The final version shown on GitHub is the submitted version. If its GitHub timestamp is after the deadline, the submission receives 0 points. No late-submission exceptions are allowed.

For the required format and GitHub submission steps, see Discussion Questions: Procedures and Rules.

NoteDiscussion session

The Week 5 discussion session is Tuesday, September 29, 2026. Open Popping when the instructor asks you to join.

See the Discussion tab for login information. Tell the instructor during class if you cannot sign in. There is no quiz during this session.

Quiz 2 is Tuesday, October 6, 2026. It covers the Week 3 through Week 5 discussions, including Week 5. See Quizzes for the quiz format.

Discussion Question Topics

The following topics are suggestions. Your question is not limited to this list, but it should relate to the Week 5 material.

  1. Alternative distance measures. Distances beyond those covered in the lecture, their suitability for different types of data, and how they express different notions of similarity.

  2. Redundant predictors and variable weighting. How repeated or strongly correlated measurements influence distance and neighbor selection, even when they add little information.

  3. Variable selection for KNN. The benefits and limitations of selecting predictors before applying KNN, including connections to lasso and the distinction between linear and nonlinear predictive relationships.

  4. Data representation and meaningful similarity. How encoding, transformations, and preprocessing affect which observations appear similar, especially for images, categorical variables, and mixed data.

  5. Dimensionality and predictive information. The distinction between the number of measured variables, the underlying dimension of the data, and the information needed to predict the response.

  6. Bias-variance trade-offs across models. How KNN’s bias-variance trade-off compares with those in subset selection, ridge regression, and lasso, including the sources of bias and how each method controls model flexibility.

  7. Agentic AI utilization issues. Specific difficulties encountered when using an agent to implement, compare, or interpret KNN analyses. Submissions should describe concrete examples rather than general opinions about AI.

NoteChoosing a question

Ask one focused question that calls for an explanation, a comparison, a short derivation, a counterexample, a small computation, or a diagnosis of a concrete difficulty. Do not submit a request for code alone or repeat a lecture calculation without adding a question about the statistical reasoning or the specific difficulty you encountered.

Week 5 Check List