Skip to main content

Evaluating Classification Models

Confusion matrices, decision cutoffs, and ROC curves

On this page

• Learning goals
• Start with estimated probabilities
• Count decisions with a confusion matrix
• Sensitivity and specificity
• Move the cutoff
• From moving cutoffs to an ROC curve
• Summarize the curve with AUC
• A few practical considerations
• Check your understanding
• Key ideas
• References and further reading

← Week 6 overview · Previous: Probabilities, Decisions, and Learning Error

A fitted logistic regression gives an estimated probability for each observation. A practitioner still needs to decide which probabilities are high enough to call positive. Lowering that cutoff finds more positive cases, but also creates more false positives. We will use a confusion matrix to count these consequences and an ROC curve to display the trade-off over all cutoffs.

Learning goals

By the end of this lecture, you should be able to:

• obtain estimated probabilities from a fitted logistic regression;
• turn those probabilities into decisions at a chosen cutoff;
• construct a confusion matrix and calculate sensitivity and specificity;
• explain what changes when the cutoff moves; and
• plot an ROC curve and interpret its AUC.

Start with estimated probabilities

As in the previous lecture, let $p(x)=P(Y=1\mid X=x)$ and let $\widehat p(x)$ be its fitted estimate. Class 1 is the positive class. The cutoff $t$ gives the decision

$$\widehat d(x)=I\{\widehat p(x)\geq t\}.$$

For example, an estimated probability of $0.30$ gives class 0 at cutoff $0.50$ and class 1 at cutoff $0.20$. The estimated probability has not changed. The decision has.

Guiding question. As we lower the cutoff, how many more positive cases do we find, and how many negative cases do we incorrectly call positive?

Fit a logistic regression

We use the same simulated two-predictor dataset throughout this lecture. It contains 600 training observations and 600 independent test observations. Both languages read the same supplied file and use its existing split. Download the classification data or run the code from the course repository root.

Logistic regression models the log odds as a linear function of the predictors. For our two predictors,

$$\log\frac{p(x)}{1-p(x)}=\beta_0+\beta_1x_1+\beta_2x_2.$$

After fitting the coefficients, the estimated probability is

$$\widehat p(x)= \frac{\exp(\widehat\beta_0+\widehat\beta_1x_1+\widehat\beta_2x_2)} {1+\exp(\widehat\beta_0+\widehat\beta_1x_1+\widehat\beta_2x_2)}.$$

This transformation keeps the prediction between zero and one. We fit the coefficients by minimizing training log loss, introduced in the previous lecture. We use ordinary logistic regression with an intercept and no penalty. The package handles the numerical fitting so that we can concentrate on evaluating the predictions.

Run the blocks in your chosen language in order. In R, type = "response" requests probabilities. In Python, predict_proba returns one column per class; the column indexed by 1 gives the probability of class 1 for our labels 0 and 1.

R

data <- read.csv("data/week-06/classification.csv")
train <- subset(data, split == "train")
test <- subset(data, split == "test")

fit <- glm(y ~ x1 + x2, data = train, family = binomial())
p_hat <- predict(fit, newdata = test, type = "response")
y <- test$y
round(coef(fit), 3)

(Intercept)          x1          x2 
     -1.306       1.080      -0.376 

Python

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.linear_model import LogisticRegression

data = pd.read_csv("data/week-06/classification.csv")
train = data.loc[data["split"] == "train"]
test = data.loc[data["split"] == "test"]

# Infinite C removes the coefficient penalty.
fit = LogisticRegression(C=np.inf, solver="lbfgs", tol=1e-10, max_iter=1000)
fit.fit(train[["x1", "x2"]], train["y"])
p_hat = fit.predict_proba(test[["x1", "x2"]])[:, 1]
y = test["y"].to_numpy()
print(np.round(np.r_[fit.intercept_, fit.coef_[0]], 3))

[-1.306  1.08  -0.376]

The fitted coefficients are approximately $-1.306$, $1.080$, and $-0.376$. Thus, holding $x_2$ fixed, a larger $x_1$ gives a larger fitted probability. At $x_1=x_2=0$, the fitted probability is approximately $0.213$.

In the previous lecture, changing the training sample changed the fitted probabilities. Here, the fitted coefficients and all 600 test probabilities stay fixed. We change only the cutoff, which changes the decisions made from those probabilities.

Count decisions with a confusion matrix

A confusion matrix compares predicted classes with observed classes. Here, rows are observed classes and columns are predicted classes.¹

	Predict 0	Predict 1
Observed 0	True negative (TN)	False positive (FP)
Observed 1	False negative (FN)	True positive (TP)

A false positive is an observed class-0 case that we call positive. A false negative is an observed class-1 case that we miss. Adding FP and FN gives the number of mistakes counted by the previous lecture’s $0$-$1$ loss. Start with cutoff $t=0.50$:

R

t <- 0.50
predicted <- as.integer(p_hat >= t)
table(Observed = factor(y, levels = c(0, 1)),
      Predicted = factor(predicted, levels = c(0, 1)))

        Predicted
Observed   0   1
       0 407  18
       1 121  54

Python

from sklearn.metrics import confusion_matrix

t = 0.50
predicted = (p_hat >= t).astype(int)
counts = pd.DataFrame(confusion_matrix(y, predicted, labels=[0, 1]),
                      index=[0, 1], columns=[0, 1])
counts.index.name = "Observed"
counts.columns.name = "Predicted"
print(counts)

Predicted    0   1
Observed          
0          407  18
1          121  54

There are 425 observed negatives and 175 observed positives. At cutoff $0.50$, the fitted rule correctly calls 407 of the negatives and 54 of the positives. It makes 18 false-positive and 121 false-negative decisions.

Sensitivity and specificity

Sensitivity measures how often we detect an observed positive. Specificity measures how often we correctly reject an observed negative:

$$\text{Sensitivity}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}, \qquad \text{Specificity}=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}}.$$

The denominators are different. Sensitivity uses only actual positives; specificity uses only actual negatives. Sensitivity is also called the true-positive rate. The false-positive rate is

$$\text{False-positive rate}=\frac{\mathrm{FP}}{\mathrm{TN}+\mathrm{FP}} =1-\text{Specificity}.$$

Keep this complement in mind: the ROC curve below will use the false-positive rate on its horizontal axis and sensitivity on its vertical axis.

For our cutoff of $0.50$,

$$\text{Sensitivity}=\frac{54}{175}\approx0.309, \qquad \text{Specificity}=\frac{407}{425}\approx0.958.$$

The rule correctly rejects most negative cases, but detects fewer than one-third of the positive cases. Reporting these two rates makes that trade-off visible.

Move the cutoff

Lower the cutoff from $0.50$ to $0.35$. Observations with fitted probabilities between $0.35$ and $0.50$ now change from predicted class 0 to predicted class 1. No observation moves in the other direction.

Consequently, lowering the cutoff can only increase or leave unchanged the true-positive count and the false-positive count. Sensitivity rises or stays the same, while specificity falls or stays the same.

Compare three cutoffs

Use $t=0.20$, $0.35$, and $0.50$. These are illustrative choices, not cutoffs selected by optimizing test performance. The following code calculates the four counts directly from the decisions.

R

cutoffs <- c(0.20, 0.35, 0.50)
rates <- data.frame(cutoff = cutoffs, TN = 0, FP = 0, FN = 0, TP = 0,
                    sensitivity = 0, specificity = 0)
for (i in seq_along(cutoffs)) {
  predicted <- p_hat >= cutoffs[i]
  TN <- sum(!predicted & y == 0)
  FP <- sum(predicted & y == 0)
  FN <- sum(!predicted & y == 1)
  TP <- sum(predicted & y == 1)
  rates[i, -1] <- c(TN, FP, FN, TP, TP / (TP + FN), TN / (TN + FP))
}
print(round(rates, 3), row.names = FALSE)

 cutoff  TN  FP  FN  TP sensitivity specificity
   0.20 262 163  41 134       0.766       0.616
   0.35 361  64  78  97       0.554       0.849
   0.50 407  18 121  54       0.309       0.958

Python

cutoffs = [0.20, 0.35, 0.50]
rows = []
for t in cutoffs:
    predicted = p_hat >= t
    TN = np.sum(~predicted & (y == 0))
    FP = np.sum(predicted & (y == 0))
    FN = np.sum(~predicted & (y == 1))
    TP = np.sum(predicted & (y == 1))
    rows.append([t, TN, FP, FN, TP, TP / (TP + FN), TN / (TN + FP)])
rates = pd.DataFrame(rows, columns=["cutoff", "TN", "FP", "FN", "TP",
                                    "sensitivity", "specificity"])
print(rates.round(3).to_string(index=False))

 cutoff  TN  FP  FN  TP  sensitivity  specificity
   0.20 262 163  41 134        0.766        0.616
   0.35 361  64  78  97        0.554        0.849
   0.50 407  18 121  54        0.309        0.958

At cutoff $0.35$, sensitivity rises to $97/175\approx0.554$ and specificity falls to $361/425\approx0.849$. Compared with $0.50$, we detect 43 additional positive cases and incorrectly call 46 additional negative cases positive.

At cutoff $0.20$, sensitivity rises further to $134/175\approx0.766$, while specificity falls to $262/425\approx0.616$. The lower cutoff finds more positives at the cost of more false alarms.

See which probabilities cross the cutoff

The next panels show the fitted probabilities separately for the two observed classes. The distributions are identical in all three panels. Only the vertical cutoff moves. For the observed positives, probabilities to the right of the cutoff are true positives. For the observed negatives, probabilities to the right are false positives.

R

breaks <- seq(0, 1, by = 0.05)
negative <- hist(p_hat[y == 0], breaks = breaks, plot = FALSE)
positive <- hist(p_hat[y == 1], breaks = breaks, plot = FALSE)
top <- max(negative$density, positive$density) * 1.25
old_par <- par(mfrow = c(1, 3), mar = c(4, 4, 3, 1))
for (t in rev(cutoffs)) {
  plot(negative, freq = FALSE, col = adjustcolor("deepskyblue", 0.45),
       border = "white", xlim = c(0, 1), ylim = c(0, top),
       xlab = "Estimated probability", ylab = "Density within class",
       main = sprintf("Cutoff = %.2f", t))
  plot(positive, freq = FALSE, col = adjustcolor("darkorange", 0.45),
       border = "white", add = TRUE)
  abline(v = t, lty = 2, lwd = 2)
  if (t == 0.50) {
    legend("topright", c("Observed 0", "Observed 1"),
           fill = c("deepskyblue", "darkorange"), bty = "n", cex = 0.8)
  }
}

The same logistic probabilities at three cutoffs. Values at or above the dashed line are classified as positive.

par(old_par)

Python

bins = np.linspace(0, 1, 21)
fig, axes = plt.subplots(1, 3, figsize=(12, 4), sharey=True)
for ax, t in zip(axes, reversed(cutoffs)):
    ax.hist(p_hat[y == 0], bins=bins, density=True, alpha=0.45,
            color="deepskyblue", edgecolor="white", label="Observed 0")
    ax.hist(p_hat[y == 1], bins=bins, density=True, alpha=0.45,
            color="darkorange", edgecolor="white", label="Observed 1")
    ax.axvline(t, color="black", linestyle="--", linewidth=2)
    ax.set(xlabel="Estimated probability", title=f"Cutoff = {t:.2f}",
           xlim=(0, 1))
axes[0].set_ylabel("Density within class")
axes[0].legend(loc="upper right", frameon=False, fontsize=8)
fig.tight_layout()
plt.show()

The same logistic probabilities at three cutoffs. Values at or above the dashed line are classified as positive.

Each class histogram has total area one, so we can compare the distributions even though there are more negatives than positives. As the cutoff moves left, the fraction of the orange distribution to its right grows, increasing sensitivity. The fraction of the blue distribution to its left shrinks, decreasing specificity. The confusion matrices above give the exact counts.

From moving cutoffs to an ROC curve

The receiver operating characteristic (ROC) curve extends our three-cutoff comparison to every cutoff. Each confusion matrix supplies a sensitivity and a false-positive rate, which become the coordinates of one point. Its axes are

$$\text{horizontal axis}=1-\text{Specificity}, \qquad \text{vertical axis}=\text{Sensitivity}.$$

Each cutoff gives one point. To construct the curve:

1. Start above the largest fitted probability. No observation is called positive, so both rates are zero.
2. Lower the cutoff through the fitted probabilities. Each newly included positive increases the true-positive rate; each newly included negative increases the false-positive rate.
3. Finish at or below the smallest fitted probability. Everyone is called positive, so both rates are one.

The curve therefore runs from $(0,0)$ to $(1,1)$ as the cutoff decreases. A useful model moves upward quickly while accumulating relatively few false positives. The upper-left corner represents sensitivity one and specificity one.

Plot the ROC curve

In R, the loop below applies the same counting calculation at every distinct fitted probability. In Python, roc_curve performs that calculation directly. Both use the estimated probabilities as their input, rather than the class labels from a single cutoff.²

R

thresholds <- c(Inf, sort(unique(p_hat), decreasing = TRUE))
tpr <- fpr <- numeric(length(thresholds))
for (i in seq_along(thresholds)) {
  predicted <- p_hat >= thresholds[i]
  tpr[i] <- mean(predicted[y == 1])
  fpr[i] <- mean(predicted[y == 0])
}

plot(fpr, tpr, type = "l", col = "deepskyblue", lwd = 3,
     xlim = c(0, 1), ylim = c(0, 1), asp = 1,
     xlab = "False-positive rate (1 - specificity)",
     ylab = "True-positive rate (sensitivity)")
abline(0, 1, col = "gray60", lty = 2)
points(1 - rates$specificity, rates$sensitivity,
       pch = 19, col = "darkorange", cex = 1.2)
text(1 - rates$specificity, rates$sensitivity,
     labels = sprintf("t = %.2f", rates$cutoff), pos = 4, cex = 0.85)

ROC curve for the fixed logistic regression on 600 test observations. The marked points use the same three cutoffs as the confusion matrices.

Python

from sklearn.metrics import roc_curve

fpr, tpr, thresholds = roc_curve(y, p_hat, drop_intermediate=False)

fig, ax = plt.subplots(figsize=(6, 6))
ax.plot(fpr, tpr, color="deepskyblue", linewidth=3)
ax.plot([0, 1], [0, 1], color="gray", linestyle="--")
ax.scatter(1 - rates["specificity"], rates["sensitivity"],
           color="darkorange", s=45, zorder=3)
for row in rates.itertuples():
    ax.annotate(f"t = {row.cutoff:.2f}",
                (1 - row.specificity, row.sensitivity),
                xytext=(7, 0), textcoords="offset points", fontsize=9)
ax.set(xlim=(0, 1), ylim=(0, 1),
       xlabel="False-positive rate (1 - specificity)",
       ylabel="True-positive rate (sensitivity)")
ax.set_aspect("equal")
fig.tight_layout()
plt.show()

ROC curve for the fixed logistic regression on 600 test observations. The marked points use the same three cutoffs as the confusion matrices.

The point for $t=0.50$ has a low false-positive rate but also low sensitivity. Moving to $t=0.35$ and then $t=0.20$ moves the point upward and to the right. This is exactly the trade-off in the confusion matrices, displayed on common axes.

The diagonal describes a rule that selects the same fraction of each class, as random selection would on average. A curve above it indicates useful ranking: positive cases tend to receive larger fitted probabilities than negative cases.

Summarize the curve with AUC

The area under the ROC curve, or AUC, summarizes how well the fitted probabilities rank the two classes. It ranges from zero to one:

• AUC $=1$ means every positive case is ranked above every negative case.
• AUC $=0.5$ is the benchmark for scores that do not distinguish the classes.
• AUC below $0.5$ indicates ranking that favors negatives over positives on the evaluated sample.

Compute the area with trapezoids in R or roc_auc_score in Python:

R

auc <- sum(diff(fpr) * (head(tpr, -1) + tail(tpr, -1)) / 2)
round(auc, 3)

[1] 0.791

Python

from sklearn.metrics import roc_auc_score

auc = roc_auc_score(y, p_hat)
print(round(auc, 3))

0.791

The test AUC is approximately $0.791$. Equivalently, if we choose one positive and one negative from this test sample, the positive has the larger fitted probability about 79.1 percent of the time, counting ties as half a success.

AUC does not mean that 79.1 percent of class decisions are correct. Class decisions require a cutoff; AUC summarizes ranking across all cutoffs. Moving the cutoff changes the confusion matrix and the marked point, but leaves the ROC curve and its AUC unchanged.

A few practical considerations

Choose the cutoff for the decision

There is no universally best point on an ROC curve. If missing a positive case is especially costly, a lower cutoff may be appropriate. If false alarms are especially costly, a higher cutoff may be appropriate. The previous lecture derived a cutoff from the relative costs when true probabilities are available.

When choosing a cutoff from observed performance, use validation data or cross-validation within the training data. Then fix the model and cutoff before reporting final test performance. The several test cutoffs in this lecture illustrate the trade-off; they are not a procedure for choosing the best cutoff on the test set.

Interpret the evaluation sample

This test set is imbalanced: 175 of its 600 cases are positive. Always predicting class 0 would be correct for 70.8 percent of cases, yet have zero sensitivity. This explains why separate summaries for the two classes can be more informative than an overall percentage correct.

All reported rates and the AUC are estimates from 600 test observations. A different test sample would give different numbers. Also, a useful AUC does not establish that a fitted probability of $0.8$ corresponds to an 80 percent event rate. Ranking and numerical probability accuracy answer different questions.

About the simulated data

The supplied data use independent standard normal predictors and Bernoulli outcomes with

$$p(x)=\frac{\exp\{-1.6+1.5x_1-x_2+1.2\sin(1.2x_1x_2)\}} {1+\exp\{-1.6+1.5x_1-x_2+1.2\sin(1.2x_1x_2)\}}.$$

Our logistic fit uses only the linear terms, so it is a working approximation to this population probability. The confusion matrix and ROC calculations do not require the model to be exactly correct. The file’s true_probability column records the simulation truth; neither fitting nor evaluation uses it. We also leave the supplied fold column unused because this example does not tune the fitted model.

Check your understanding

1. A case has fitted probability $0.30$. What happens to its predicted class when the cutoff changes from $0.50$ to $0.20$? What else is needed to decide which confusion-matrix cell it belongs to?
2. Why can lowering the cutoff never reduce sensitivity on the same observations with the same fitted probabilities?
3. Explain why the ROC point moves upward and to the right as the cutoff decreases.
4. What does an AUC of $0.791$ mean? Why is it not an accuracy of 79.1 percent?
5. Why should we choose the cutoff before examining final test performance?

Key ideas

1. Logistic regression estimates probabilities; the cutoff turns them into class decisions.
2. Sensitivity describes detection among actual positives. Specificity describes correct rejection among actual negatives.
3. Lowering the cutoff finds more positives and can create more false positives.
4. The ROC curve displays this trade-off across cutoffs. AUC summarizes the ranking supplied by the fitted probabilities.
5. The appropriate cutoff depends on the decision, and final performance must be evaluated on independent observations.

References and further reading

• James, G., Witten, D., Hastie, T., Tibshirani, R., and Taylor, J. (2023). An Introduction to Statistical Learning, Chapter 4. Springer.
• The R documentation for glm describes fitting generalized linear models, including logistic regression.
• The scikit-learn documentation describes LogisticRegression, roc_curve, and roc_auc_score.

Footnotes

1. Some software and the earlier digit example place predicted classes in rows. Identify the cells from their labels, rather than assuming a fixed orientation.

2. Observations with equal fitted probabilities enter together. Connecting their ROC points by straight segments corresponds to allowing randomization within a tied group. The associated AUC gives half credit to a tied positive-negative pair. With distinct probabilities, consecutive points are joined by horizontal or vertical segments.