Week 2: Training and Test Error

This week studies why a larger linear model can fit the observed responses better while predicting less accurately. We first examine this behavior under a fixed design, then connect the theory to model-selection criteria and practical search methods.

Guiding question: If adding predictors cannot increase training MSE, why can they increase test MSE?

Learning goals

By the end of this week, you should be able to:

  • explain why training MSE is too small on average;
  • distinguish one observed test MSE from expected test MSE, also called prediction error;
  • connect the fixed-design derivation to Mallows’ CpC_p, AIC, BIC, and validation; and
  • distinguish a model-selection criterion from the algorithm used to search through models.

Lecture notes

  1. Fixed-Design Prediction Error and Model Selection

    Simulation, expected training and test MSE, and the motivation for Mallows’ CpC_p, AIC, BIC, and validation.

  2. Implementing Linear Model Selection

    Model scores, best-subset search, and stepwise search in R and Python.

Homework 02

Homework 02 contains five required problems. They move from whole-design and target-point simulations to theory, model comparison, and valid out-of-sample evaluation. All homework assignments are evaluated as Complete or Incomplete. Attempt each problem before consulting the solutions.

ImportantDeadline

Place homework-02.pdf or homework-02.html in the homework/week-02/ folder of your stat432-fall2026 GitHub repository by Sunday, September 6, 2026, at 11:59 p.m. Central Time. The report must be either a PDF or a self-contained HTML file and must not exceed 10 pages. The final file version shown on GitHub is treated as the submitted version, and its GitHub timestamp determines whether it is on time.

Discussion Session on Tuesday, September 8, 2026

ImportantSubmit Discussion Question

Submit discussion/week-02-question.md by Sunday, September 6, 2026, at 11:59 p.m. Central Time. This required weekly discussion question counts toward the Weekly GitHub Discussion Questions component, worth 10% of the course grade.

The final version shown on GitHub is the submitted version. If its GitHub timestamp is after the deadline, the submission receives 0 points. No late-submission exceptions are allowed.

For the required format and GitHub submission steps, see Discussion Questions: Procedures and Rules.

NoteDiscussion session

The Week 2 discussion session is Tuesday, September 8, 2026. Open Popping when the instructor asks you to join.

See the Discussion tab for login information. Tell the instructor during class if you cannot sign in. There is no quiz during this session.

Quiz 1 is Tuesday, September 15, 2026. It contains 8 questions and covers the Week 1 and Week 2 discussions. See Quizzes for the quiz format.

Discussion Question Topics

Your question may focus on one of the following topics:

  1. Training and test MSE. Why training MSE cannot increase across a nested sequence of least-squares models, why expected test MSE can increase, or why one observed test MSE need not follow its expectation.

  2. Approximation bias and estimation variance. How adding predictors changes mean squared approximation bias and estimation variance, and how their trade-off determines expected test MSE.

  3. Prediction at a particular target. Why the predictor count that minimizes mean squared error for estimating the mean response at one covariate value can differ from the count preferred by average test MSE over the design.

  4. Optimism and Mallows’ CpC_p. Why training error is optimistic on average, what the 2p2p term in CpC_p represents, or which assumptions support it.

  5. AIC, BIC, and model search. Why AIC and BIC can select different models, or why a model-selection criterion is distinct from best-subset or stepwise search.

  6. Honest out-of-sample evaluation. Why model selection must be repeated inside each cross-validation training fold and why final test data cannot also be used to select the model.

  7. Concrete difficulties using AI tools. A specific example from your Week 2 work in which an AI tool or agent misunderstood your request, produced incorrect or unverifiable code or results, changed the wrong file, or failed on a coding or Git task. Explain what you asked it to do, what actually happened, how you recognized the problem, and how you resolved it or where you remain stuck. Do not submit a general question about AI.

NoteChoosing a question

Ask one focused question that calls for an explanation, a comparison, a short derivation, a counterexample, a small simulation, or a diagnosis of a concrete difficulty. Do not submit a request for code alone or repeat a homework calculation without adding a question about the statistical reasoning or the difficulty you encountered.

Week 2 Check List