Week 3: Ridge Regression and Optimization

This week begins with a practical problem: ordinary least-squares coefficients can be unstable when predictors contain similar information. Ridge regression shows how shrinkage can improve stability. We then use ridge to study optimization, cross-validation, and the choice of a penalty parameter.

Guiding question: What do we gain, and what do we give up, when we deliberately shrink regression coefficients?

Learning goals

By the end of this week, you should be able to:

  • explain why correlated predictors can make least-squares coefficients unstable;
  • describe the bias and variance effects of ridge regression;
  • connect gradient descent and a direct linear-system solution to the ridge objective; and
  • compare cross-validation and generalized cross-validation (GCV) for tuning a penalty without using the final test data.

Lecture notes

  1. Ridge Regression: Stability Through Shrinkage

    Penalized least squares, spectral shrinkage, effective degrees of freedom, and the bias-variance trade-off.

  2. From a Penalized Objective to a Fitted Ridge Model

    The objective, gradient descent, step size, cross-validation and GCV, preprocessing fitted within each training fold, and held-out evaluation.

Homework 03

Homework 03 contains five required problems. Question 1 asks you to write and install a short explain-to-me skill, then test it by using the skill to understand Question 2. The remaining problems study coefficient instability under near-collinearity, singular-direction shrinkage, optimization geometry, and penalty selection by cross-validation and generalized cross-validation (GCV). All homework assignments are evaluated as Complete or Incomplete. For Questions 2 through 5, attempt each problem before consulting the solutions.

Update: The notation and wording in Homework 03 have been updated and slightly simplified. Submissions based on either version are accepted.

ImportantDeadline

Place homework-03.pdf or homework-03.html in the homework/week-03/ folder of your stat432-fall2026 GitHub repository by Sunday, September 13, 2026, at 11:59 p.m. Central Time. The report must be either a PDF or a self-contained HTML file and must not exceed 10 pages. The final file version shown on GitHub is treated as the submitted version, and its GitHub timestamp determines whether it is on time.

Discussion Session on Tuesday, September 15, 2026

ImportantSubmit Discussion Question

Submit discussion/week-03-question.md by Sunday, September 13, 2026, at 11:59 p.m. Central Time. This required weekly discussion question counts toward the Weekly GitHub Discussion Questions component, worth 10% of the course grade.

The final version shown on GitHub is the submitted version. If its GitHub timestamp is after the deadline, the submission receives 0 points. No late-submission exceptions are allowed.

For the required format and GitHub submission steps, see Discussion Questions: Procedures and Rules.

NoteQuiz 1 and discussion session

Quiz 1 begins class on Tuesday, September 15, 2026. It is 30 minutes long, contains 8 questions, and covers the Week 1 and Week 2 discussions. The Week 3 discussion session follows the quiz. Week 3 is included in Quiz 2 on Tuesday, October 6, 2026. See Quizzes for the format.

Open Popping when the instructor asks you to join. See the Discussion tab for login information, and tell the instructor during class if you cannot sign in.

Discussion Question Topics

Your question may focus on one of the following topics:

  1. Instability and shrinkage. Why correlated predictors can make ordinary least-squares coefficients unstable, and how ridge regression changes that behavior.

  2. Bias, variance, and effective degrees of freedom. What ridge regression gains and gives up as the penalty increases.

  3. Standardization and the intercept. Why predictor scaling matters and why the intercept is ordinarily left unpenalized.

  4. Optimization. How curvature and step size affect gradient descent, and why an optimization algorithm is distinct from the estimator it computes.

  5. Choosing the penalty. Why preprocessing must be repeated within each cross-validation training fold, how the minimum-error and one-standard-error rules differ, and how cross-validation and GCV compare.

  6. Concrete AI-use difficulties. A specific difficulty you encountered while using an AI tool for ridge regression, optimization, cross-validation, coding, or Git. Describe the exact prompt or task, what the tool returned or did, how you diagnosed the problem, and what remains unclear. Focus on a concrete example rather than a general opinion about AI.

  7. Agentic AI utilization issues. A specific example in which an AI agent took an action that did not work as intended, such as installing a skill incorrectly, writing to the wrong folder, activating a skill at an unwanted time, changing an unrelated file, or failing to remove a skill. Describe what you asked the agent to do, what it actually did, what evidence revealed the problem, and how you corrected it or where you remain stuck. Do not submit a general or conceptual question about agentic AI.

NoteChoosing a question

Ask one focused question that calls for an explanation, a comparison, a short derivation, a counterexample, a small computation, or a diagnosis of a concrete difficulty. Do not submit a request for code alone or repeat a homework calculation without adding a question about the statistical reasoning or the specific difficulty you encountered.

Week 3 Check List