Ridge Regression

Week 3: Sep 8 – Sep 12

Weekly Objectives

• Correlated variables and motivation of Ridge regression
• How to solve Ridge regression
• The source of bias and variance, and their trade-offs
• The effect of penalty and different ways of tuning it
• Connection between Ridge regression and PCA
• \(k\)-fold Cross-validation
• Scaling issue and other thing that could affect the performance
• Using the glmnet package and lm.ridge() from the MASS package

Lecture Notes and R Examples

• Ridge Regression and Bias-variance Trade-off [Source File]

Additional Readings

• PCA [Source File]
  
• Derivation of the Bias-variance Trade-off in a simplified case
• Ridge regression scaling issue

Homework

• Homework 3 [Source File]
• Due Sep 18, 11:59PM CT
• [Solution]