Week 6: Classification Error and Evaluation

Many classifiers first estimate a probability and then compare it with a cutoff to make a class decision. This week connects probability estimation to classification error, then uses logistic regression to study confusion matrices and ROC curves.

Guiding question: How does error in an estimated probability become error in a class decision, and what changes when we move the decision cutoff?

Learning goals

By the end of this week, you should be able to:

  • distinguish a true class probability, an estimated probability, and a class decision;
  • derive the Bayes classifier and Bayes error for binary classification;
  • connect probability losses, bias, and variance to classification error;
  • calculate sensitivity and specificity from a confusion matrix and explain how they change with the cutoff; and
  • construct an ROC curve from logistic probabilities and interpret its AUC.

Lecture notes

  1. Classification: Probabilities, Decisions, and Learning Error

    Bayes decisions and error, Brier and log loss, KNN probability bias and variance, and unequal error costs.

  2. Evaluating Classification Models

    Logistic probabilities, confusion matrices, sensitivity and specificity, changing the cutoff, and ROC curves and AUC.

Homework 06

Homework 06 compares logistic regression and KNN using ROC curves, examines training and test performance for different KNN neighborhoods, and includes peer evaluation of Homework 5.

Use either R or Python for the computational questions. All homework assignments are evaluated as Complete or Incomplete. Attempt each computational question before consulting the solutions. Complete Question 4, part 2, yourself without AI assistance.

NoteDraft homework

Questions 1, 2, and 4 are drafted. Questions 3 and 5 and the computational solutions are still being prepared.

ImportantDeadline

Place homework-06.pdf or homework-06.html in the homework/week-06/ folder of your stat432-fall2026 GitHub repository by Sunday, October 4, 2026, at 11:59 p.m. Central Time. The report must be either a PDF or a self-contained HTML file and must not exceed 10 pages. The final file version shown on GitHub is treated as the submitted version, and its GitHub timestamp determines whether it is on time.

Discussion Session on Tuesday, October 6, 2026

ImportantSubmit Discussion Question

Submit discussion/week-06-question.md by Sunday, October 4, 2026, at 11:59 p.m. Central Time. This required weekly discussion question counts toward the Weekly GitHub Discussion Questions component, worth 10% of the course grade.

The final version shown on GitHub is the submitted version. If its GitHub timestamp is after the deadline, the submission receives 0 points. No late-submission exceptions are allowed.

For the required format and GitHub submission steps, see Discussion Questions: Procedures and Rules.

NoteQuiz 2 and discussion

Quiz 2 takes place before the Week 6 discussion session on Tuesday, October 6, 2026. The quiz covers Weeks 3 through 5. See Quizzes for the quiz format. After the quiz, open Popping when the instructor asks you to join the discussion.

See the Discussion tab for login information. Tell the instructor during class if you cannot sign in.

Discussion Question Topics

The following topics are suggestions. Your question is not limited to this list, but it should relate to the Week 6 material or your peer review of Homework 5.

  1. Peer review of Homework 5. Mistakes or items for improvement you identified while reviewing other students’ Homework 5 reports, including code, reasoning, interpretation, or plots.

  2. Probability estimates and class decisions. Why similar probability estimates can produce different decisions, especially near the cutoff.

  3. Bayes error, bias, and variance. How unavoidable outcome variation differs from error in an estimated probability, and how each affects classification.

  4. Training and test ROC curves. How model complexity and sample size affect the gap between training and test performance.

  5. Choosing a cutoff. How sensitivity, specificity, and unequal error costs guide a decision, and why the appropriate cutoff need not be 0.5.

  6. Comparing ROC curves and AUC. What AUC summarizes and what it leaves unresolved when choosing a model or cutoff.

NoteChoosing a question

Ask one focused question that calls for an explanation, a comparison, a short derivation, a counterexample, or a small computation. Do not submit a request for code alone or repeat a homework calculation without asking about the statistical reasoning or the specific difficulty you identified.

Week 6 Check List