Homework 05

Before you begin

Homework 05 contains five required questions on KNN bias and variance, comparison with lasso, latent dimension, handwritten digit classification, and a plot-formatting skill. It is evaluated as Complete or Incomplete. Use either R or Python. You are not expected to use both languages.

Use of homework submissions

Your Homework 5 submission will be used as example material for peer evaluation and course lectures in future weeks. Use of your material will be restricted to this class and this semester only.

For Questions 1 through 4, attempt each question before looking at the solutions or asking an AI tool for help. Use an AI agent as directed in Question 5. You may use AI tools, but you are responsible for checking the work and being able to explain your reasoning, code, and results.

Download the materials

The ZIP file contains the two editable QMD files and the handwritten digit data for Python in Question 4. Generate the simulation data from the models stated below.

Set a random seed before each simulation question and report it. You may use standard packages unless a question asks you to write the calculation yourself. Use equal neighbor weights throughout. Here $p$ counts the covariates used to measure distance; do not add an intercept column to a KNN predictor matrix. R and Python use different random-number generators, so their exact simulation results need not agree.

Question 1: Estimate the bias and variance of KNN

In our previous homework, we used repeated simulations to study the behavior of linear regression estimators. We will now use the same idea for KNN. Consider the regression model

$Y_i=f(\mathbf X_i)+\epsilon_i, \qquad f(\mathbf x)=0.5x_1+\sin(x_2)-0.3x_3^2,$

where the three covariates are independent standard normal variables and $\epsilon_i\sim\mathcal N(0,0.1^2)$ independently of the covariates and other observations. Generate $n=400$ training observations. Our goal is to estimate the mean response at

$\mathbf x_0=(0.5,0.7,1)^{\mathsf T}.$

Use Euclidean distance and consider $k=1,3,5,\ldots,29$ .

Calculate $f(\mathbf x_0)$ . Generate one training dataset and obtain the KNN prediction at $\mathbf x_0$ for each value of $k$ .
Independently repeat the generation of the training covariates and responses 200 times. Within each repetition, use the same training data for every value of $k$ . Store the predictions in a matrix with 200 rows and one column for each value of $k$ . Use this matrix to estimate the bias, variance, and mean squared error of the estimated mean response. Use divisor 200 when calculating the empirical variance, and verify numerically that

$\text{mean squared error} =\text{squared bias}+\text{variance}.$

Plot squared bias, variance, and mean squared error against $k$ in one figure. Explain the pattern and compare it with the bias-variance trade-off for ridge regression or lasso.

Question 2: Compare KNN with lasso

Use the response model from Question 1, including the noise standard deviation $0.1$ , but generate $p=30$ covariates. The response still depends only on the first three. Consider these two settings:

Setting 1: all 30 covariates are independent standard normal variables.
Setting 2: the covariates have a multivariate normal distribution with mean zero and covariance matrix $\boldsymbol\Sigma$ , where

$\Sigma_{jj}=1, \qquad \Sigma_{jl}=0.8\quad\text{for }j\ne l.$

In each setting, generate 400 training observations and 1,000 independent test observations. Fit 5NN using Euclidean distance on the supplied covariate scale. All covariates already have population variance one.

For lasso, use a linear model containing the 30 covariates and a separately fitted, unpenalized intercept. Follow the Week 4 objective,

$L_\lambda(\boldsymbol\beta) =\frac{1}{2n}\left\lVert \widetilde{\mathbf y}-\mathbf X\boldsymbol\beta \right\rVert_2^2 +\lambda\lVert\boldsymbol\beta\rVert_1,$

where $\mathbf X\in\mathbb R^{n\times30}$ contains the centered and standardized covariates and $\widetilde{\mathbf y}$ is the centered training response. Select the penalty by ten-fold cross-validation over 41 logarithmically spaced values from $10^{-4}$ to $1$ . Estimate all means and scales using only the training portion of each fold, with divisor equal to that portion’s sample size. After selecting the penalty, refit on all training observations and apply the fitted transformations unchanged to the test observations. Do not add nonlinear terms to the lasso model.

Fit both methods in each setting. Use the same training and test data for the two methods within a setting. Report their test MSE values in one table, together with the selected lasso penalty and the number of nonzero lasso slopes. You only need to generate one dataset for each setting; repeated simulations are not required.
Compare the results between the two settings. Explain how correlation can change the usefulness of Euclidean neighborhoods, and why lasso does not use all covariates in the same way as KNN.
The response model contains $\sin(X_2)$ and $X_3^2$ , but the lasso fit uses only linear terms. Explain how this difference affects the comparison. Would the results justify a general claim that correlation always helps KNN, or that one method is always better? Explain why.

Question 3: Observed dimension and latent dimension

The lecture’s handwritten digit example shows that KNN can work with many measured covariates. We will use a simulation to investigate why the structure of those covariates matters.

Let $p=100$ be the number of observed covariates and let $m$ be the number of latent variables. For each observation, generate

$\mathbf X_i=\mathbf A\mathbf Z_i+\boldsymbol\eta_i, \qquad Y_i=Z_{i1}+Z_{i2}+\epsilon_i,$

where $\mathbf A\in\mathbb R^{100\times m}$ has independent entries from Uniform $[-2,2]$ . The entries of $\mathbf Z_i\in\mathbb R^m$ and $\boldsymbol\eta_i\in\mathbb R^{100}$ , and the scalar $\epsilon_i$ , are independent standard normal variables, also independent of $\mathbf A$ . Generate $\mathbf A$ once and use it for all training and test observations in that dataset.

Compare $m=3$ and $m=30$ . In each setting, generate 200 training observations and 200 independent test observations. Fit KNN using only the 100 observed covariates in $\mathbf X$ , with Euclidean distance on their generated scale. The latent variables in $\mathbf Z$ are used to generate the data but are not available to the fitted model. Consider $k=2,10,18,\ldots,82$ .

Generate one dataset for each value of $m$ . Fit KNN for every value of $k$ and calculate test MSE. Within a setting, use the same training and test observations for all values of $k$ .
Repeat the complete experiment 50 times, generating a new $\mathbf A$ and new observations in each repetition. Plot average test MSE against $k$ , with one curve for each value of $m$ . These curves compare the specified choices of $k$ in a controlled simulation; they are not a cross-validation procedure for selecting a final model.
Both settings have 100 observed covariates, and the response depends on only two latent variables. Explain why their KNN performance can still differ. What role does the noise $\boldsymbol\eta_i$ play, and why are the observed data not confined exactly to an $m$ -dimensional linear space?

Question 4: Classify handwritten digits

Use zip.train and zip.test from the R package ElemStatLearn, as in the lecture. The package has been retired from the active CRAN repository. If needed, install the archived package into your usual R library, then load the two datasets. Python users can use the course copies of the same training data and test data. These compressed CSV files have no headers and are included in the homework ZIP under data/knn. Run the Python code from the folder containing data.

Each image has $16\times16=256$ pixel values. The first column is the digit label and the remaining columns are the pixel values. Use the first 1,000 rows of zip.train and the first 500 rows of zip.test. Use the original pixel values without standardization, and do not include the digit label in a distance calculation.

For a test image $\mathbf x_0$ and a training image $\mathbf x_i$ , the two distances are

$d_E(\mathbf x_0,\mathbf x_i) =\sqrt{\sum_{j=1}^{256}(x_{0j}-x_{ij})^2}, \qquad d_M(\mathbf x_0,\mathbf x_i) =\sum_{j=1}^{256}|x_{0j}-x_{ij}|.$

Write a 5NN classification function that allows either Euclidean or Manhattan distance. For each test image, calculate its distances to the training images directly, find the five nearest, and return the most frequent digit label. Break distance ties by the smaller training row number and voting ties by the smallest digit. Do not use a built-in distance or KNN function for this part.
Apply the classifier to all 500 test images using each distance. For each fit, report the test classification error and a $10\times10$ confusion matrix, with rows representing predicted digits and columns representing true digits. Keep all digit labels from 0 through 9 in the matrix, including any with a zero count. Which digits are most often confused with one another?
For Euclidean distance, display three incorrectly classified test images, together with their true and predicted labels. Comment on whether the mistakes are visually understandable. Compare the overall results of the two distance measures. Can they have similar total error while making mistakes on different images? Use your predictions to support your answer.

Question 5: Create a plot-formatting skill

Create a short SKILL.md file for improving the formatting of STAT 432 homework plots. You may use AI to help write it.

Include a title, a description, and a few formatting rules. The description should limit the skill to STAT 432 homework plot improvements and require an explicit request to use it. Your rules should address margins, titles, axes, labels, colors, and sizes for plots produced in R or Python. Keep them portable: do not refer to specific homework questions.
Ask your agent to install the skill. Explicitly call it to improve the plot from Question 1, then compare the original and revised versions. Do you like the changes? You may revise the skill and test it a couple more times, or uninstall it if it remains unhelpful.
Append your final SKILL.md content to your submission and briefly explain your choices and experience.