Stat 432 Homework 8

Instruction

Please remove this section when submitting your homework.

Students are encouraged to work together on homework and/or utilize advanced AI tools. However, sharing, copying, or providing any part of a homework solution or code to others is an infraction of the University’s rules on Academic Integrity. Any violation will be punished as severely as possible. Final submissions must be uploaded to Gradescope. No email or hard copy will be accepted. For late submission policy and grading rubrics, please refer to the course website.

• You are required to submit the rendered file HWx_yourNetID.pdf. For example, HW01_rqzhu.pdf. Please note that this must be a .pdf file. .html format cannot be accepted. Make all of your R code chunks visible for grading.
• Include your Name and NetID in the report.
• If you use this file or the example homework .Rmd file as a template, be sure to remove this instruction section.
• Make sure that you set seed properly so that the results can be replicated if needed.
• For some questions, there will be restrictions on what packages/functions you can use. Please read the requirements carefully. As long as the question does not specify such restrictions, you can use anything.
• When using AI tools, you are encouraged to document your comment on your experience with AI tools especially when it’s difficult for them to grasp the idea of the question.
• On random seed and reproducibility: Make sure the version of your R is \(\geq 4.0.0\). This will ensure your random seed generation is the same as everyone else. Please note that updating the R version may require you to reinstall all of your packages.

Question 1: Linear Discriminant Analysis

Load the MNIST dataset, the same way as previous HW.

  # readin the data
  # mnist <- read.csv("https://pjreddie.com/media/files/mnist_train.csv", nrows = 2000)
  # colnames(mnist) = c("Digit", paste("Pixel", seq(1:784), sep = ""))
  # save(mnist, file = "mnist_first2000.RData")

  # you can load the data with the following code
  load("mnist_first2000.RData")
  dim(mnist)

## [1] 2000  785

1. We aim to fit an LDA (Linear Discriminant Analysis) model with our own defined function following our understanding of the LDA. An issue with this dataset, as we saw earlier, is that some pixels display little or no variation across all observations. This zero variance issue poses a problem when inverting the estimated covariance matrix. Do the following to address this issue and fit the LDA model.

   • Within the first 1000 observations, extract digts 1, 6, and 7 as the training dataset. Do the same for the second 1000 observations as the testing dataset.
   • To remove variables (pixels) with low variance, do the same procedure as we did before to select the top 300 Pixels with the highest variance.
   • Following our lecture note, the LDA model requires the estimation of several quantities, \(\Sigma\), \(\mu_k\) and \(\pi_k\). Perform these estimations and calculate the linear decision score \(x^T w_k + b_k\).
   • Use the linear decision score to predict the class label on the testing data. Report the prediction error and the confusion matrix based on the testing data.

2. The result was not ideal. At least compared with our previous HW using SVM one-vs-one model, this is probably worse. Let’s try to improve it. One issue could be that the inverse of the covariance matrix is not very stable. As we discussed in class, one possible choice is to add a ridge penalty to \(\Sigma\). Carry out this approach using \(\lambda = 1\) and re-calculate the confusion matrix and prediction error. Then try a few different penalty values of \(\lambda\) to observe how the prediction error changes. Comment on the effect of \(\lambda\), specifically under the context of this model.

3. Another approach we could do is to perform PCA at the very beginning of this analysis, instead of screening for the top 300 variables. And then we can perform the same type of analysis as in part a. but with PCA as your variables (in both training and testing data).

   • Start with the original complete mnist data, and take digits 1, 6, and 7. Perform PCA on the pixels.
   • Take the first 20 PCs as your new dataset, and then split the data back to training and testing data, with the same sample size as part a).
   • Perform the same analysis as in part a. and report the confusion matrix and prediction error.

   Comment on why do you think this approach would work well.