Support Vector Regression
Ruoqing Zhu
Last Updated: August 23, 2026
1 Support Vector Regression
Support Vector Regression (SVR) extends support vector machines to regression problems. This lecture assumes familiarity with Support Vector Machines, especially margins, slack variables, Lagrange duality, and support vectors. Unlike SVM for classification, which tries to maximize the margin between classes, SVR fits a regression function while ignoring deviations within a margin of tolerance, called the \(\epsilon\)-insensitive tube. Following the same logic as SVM, we will start from the primal formulation and then derive the dual. Finally, using the RKHS framework, we will extend SVR to nonlinear regression problems.
2 The \(\epsilon\)-insensitive Loss
For an observed dataset \(\{(\bx_i,y_i)\}_{i=1}^n\), where \(\bx_i\in\RR^p\), a linear SVR model aims to find a prediction rule
\[ f(\bx)=\beta_0+\bbeta^\T\bx, \]
such that deviations smaller than \(\epsilon\) are ignored. Formally, the \(\epsilon\)-insensitive loss is
\[\begin{align} L_\epsilon(y,f(\bx)) &= \begin{cases} 0, & \text{if } |y-f(\bx)|\leq\epsilon,\\ |y-f(\bx)|-\epsilon, & \text{otherwise}, \end{cases} \nonumber\\ &=\max\{0,|y-f(\bx)|-\epsilon\}. \end{align}\]
This loss function can be visualized as follows:
epsilon_loss <- function(y, f, epsilon) {
pmax(0, abs(y - f) - epsilon)
}
y_vals <- seq(-2.5, 2.5, length.out = 100)
f_vals <- 0
plot(y_vals, epsilon_loss(y_vals, f_vals, 1),
type = "l", col = "blue", lwd = 2,
ylim = c(0, 2.5),
xlab = "y - f(x)", ylab = "Loss",
main = expression(epsilon*"-insensitive Loss Function"))
abline(v = c(-1, 1), col = "gray50", lty = 3)
The loss is exactly zero inside the tube and grows linearly once the absolute residual exceeds \(\epsilon\). Therefore, observations well inside the tube do not directly change the empirical loss.
3 Primal and Dual Formulation of SVR
The primal optimization problem for SVR allows up to \(\epsilon\) deviation from the observed \(y_i\) values and penalizes larger deviations. We introduce slack variables \(\xi_i\) and \(\xi_i'\): \(\xi_i\) handles \(y_i-f(\bx_i)>\epsilon\), while \(\xi_i'\) handles \(f(\bx_i)-y_i>\epsilon\). The primal problem is
\[\begin{align} \min_{\beta_0,\bbeta,\bxi,\bxi'}\quad &\frac{1}{2}\|\bbeta\|^2+C\sum_{i=1}^n(\xi_i+\xi_i')\\ \text{subject to}\quad &y_i-(\beta_0+\bbeta^\T\bx_i)\leq\epsilon+\xi_i,\\ &(\beta_0+\bbeta^\T\bx_i)-y_i\leq\epsilon+\xi_i',\\ &\xi_i,\xi_i'\geq0,\qquad i=1,\ldots,n. \end{align}\]
Here \(C>0\) controls the tradeoff between a flat regression function and violations outside the \(\epsilon\)-tube. A larger \(C\) penalizes outside-tube deviations more strongly, while \(\epsilon\) determines the width of the unpenalized tube.
To derive the dual formulation, introduce nonnegative Lagrange multipliers \(\alpha_i,\alpha_i',\eta_i,\eta_i'\geq0\) for the constraints corresponding to each observation. The Lagrangian is
\[ \begin{aligned} \cL(\beta_0,\bbeta,\bxi,\bxi',\balpha,\balpha',\boldsymbol{\eta},\boldsymbol{\eta}') ={}&\frac{1}{2}\|\bbeta\|^2+C\sum_{i=1}^n(\xi_i+\xi_i')\\ &-\sum_{i=1}^n\alpha_i\big[\epsilon+\xi_i-y_i+(\beta_0+\bbeta^\T\bx_i)\big]\\ &-\sum_{i=1}^n\alpha_i'\big[\epsilon+\xi_i'+y_i-(\beta_0+\bbeta^\T\bx_i)\big]\\ &-\sum_{i=1}^n\eta_i\xi_i-\sum_{i=1}^n\eta_i'\xi_i'. \end{aligned} \]
We take partial derivatives with respect to the primal variables and set them to zero:
\[ \begin{aligned} \frac{\partial\cL}{\partial\bbeta} &=\bbeta-\sum_{i=1}^n(\alpha_i-\alpha_i')\bx_i=\mathbf{0},\\ \frac{\partial\cL}{\partial\beta_0} &=-\sum_{i=1}^n(\alpha_i-\alpha_i')=0,\\ \frac{\partial\cL}{\partial\xi_i} &=C-\alpha_i-\eta_i=0,\\ \frac{\partial\cL}{\partial\xi_i'} &=C-\alpha_i'-\eta_i'=0. \end{aligned} \]
Hence,
\[ \bbeta=\sum_{i=1}^n(\alpha_i-\alpha_i')\bx_i, \qquad \sum_{i=1}^n(\alpha_i-\alpha_i')=0, \qquad 0\leq\alpha_i,\alpha_i'\leq C. \]
The primal problem is convex and can be made strictly feasible by choosing sufficiently large slack variables, so strong duality holds. Substituting the stationarity conditions into the Lagrangian gives the dual objective
\[ \begin{aligned} g(\balpha,\balpha') ={}&\frac{1}{2}\sum_{i=1}^n\sum_{j=1}^n (\alpha_i-\alpha_i')(\alpha_j-\alpha_j') \langle\bx_i,\bx_j\rangle\\ &-\sum_{i=1}^n\alpha_i\left( \epsilon-y_i+ \sum_{j=1}^n(\alpha_j-\alpha_j')\langle\bx_j,\bx_i\rangle \right)\\ &-\sum_{i=1}^n\alpha_i'\left( \epsilon+y_i- \sum_{j=1}^n(\alpha_j-\alpha_j')\langle\bx_j,\bx_i\rangle \right)\\ ={}&-\frac{1}{2}\sum_{i=1}^n\sum_{j=1}^n (\alpha_i-\alpha_i')(\alpha_j-\alpha_j') \langle\bx_i,\bx_j\rangle\\ &-\epsilon\sum_{i=1}^n(\alpha_i+\alpha_i') +\sum_{i=1}^ny_i(\alpha_i-\alpha_i'). \end{aligned} \]
Therefore, the dual optimization problem is
\[ \begin{aligned} \max_{\balpha,\balpha'\in\RR^n}\quad &-\frac{1}{2}\sum_{i=1}^n\sum_{j=1}^n (\alpha_i-\alpha_i')(\alpha_j-\alpha_j') \langle\bx_i,\bx_j\rangle\\ &-\epsilon\sum_{i=1}^n(\alpha_i+\alpha_i') +\sum_{i=1}^ny_i(\alpha_i-\alpha_i')\\ \text{subject to}\quad &\sum_{i=1}^n(\alpha_i-\alpha_i')=0,\\ &0\leq\alpha_i,\alpha_i'\leq C,\qquad i=1,\ldots,n. \end{aligned} \]
Let \(d_i=\alpha_i-\alpha_i'\). The regression function can then be written as
\[ \begin{aligned} f(\bx) &=\beta_0+\sum_{i=1}^nd_i\langle\bx_i,\bx\rangle\\ &=\beta_0 +\sum_{i:\alpha_i>0}\alpha_i\langle\bx_i,\bx\rangle -\sum_{i:\alpha_i'>0}\alpha_i'\langle\bx_i,\bx\rangle\\ &=\beta_0+\sum_{i:\alpha_i+\alpha_i'>0} (\alpha_i-\alpha_i')\langle\bx_i,\bx\rangle. \end{aligned} \]
The KKT conditions explain the term support vector. Let \(r_i=y_i-f(\bx_i)\). If \(|r_i|<\epsilon\), then \(\alpha_i=\alpha_i'=0\). If \(r_i>\epsilon\), then \(\alpha_i=C\) and \(\alpha_i'=0\); if \(r_i<-\epsilon\), then \(\alpha_i=0\) and \(\alpha_i'=C\). Points on the boundary of the tube may have a multiplier in \([0,C]\). Thus, the observations with \(\alpha_i+\alpha_i'>0\) are the support vectors.
The intercept disappears from the dual objective because of the equality constraint, but it can be recovered from a support vector on the boundary. If \(0<\alpha_i<C\), then
\[ \beta_0 = y_i-\sum_{j=1}^nd_j\langle\bx_j,\bx_i\rangle-\epsilon. \]
If \(0<\alpha_i'<C\), then
\[ \beta_0 = y_i-\sum_{j=1}^nd_j\langle\bx_j,\bx_i\rangle+\epsilon. \]
In practice, these values can be averaged over all eligible support vectors. The e1071 package in R provides a convenient implementation of SVR.
set.seed(123)
n <- 50
x <- seq(-3, 3, length.out = n)
y <- 0.2*x + sin(x) + rnorm(n, sd = 0.2)
# fit SVR and keep epsilon on the original response scale
library(e1071)
svr.fit <- svm(
y ~ x,
cost = 10,
epsilon = 0.5,
kernel = "linear",
scale = FALSE
)
# predictions and support-vector indices
yhat <- predict(svr.fit, data.frame(x = x))
is_support <- seq_len(n) %in% svr.fit$index
outside_tube <- abs(y - yhat) > 0.5
plot(
x, y,
col = ifelse(is_support, "red", "gray50"),
pch = ifelse(is_support, 1, 19),
xlab = "X", ylab = "Y",
main = expression("SVR with "*epsilon*"-insensitive Tube"),
cex = ifelse(outside_tube, 1 + 3*(abs(y - yhat) - 0.5), 1)
)
# true function
lines(x, 0.2*x + sin(x), col = "red", lwd = 2, lty = 2)
# fitted function and epsilon tube
lines(x, yhat, col = "blue", lwd = 2)
lines(x, yhat + 0.5, col = "blue", lty = 3, lwd = 2)
lines(x, yhat - 0.5, col = "blue", lty = 3, lwd = 2)
legend(
"topleft",
legend = expression(
"Non-support Vector", "Support Vector",
"True Function", "Fitted Function", epsilon*"-Tube"
),
col = c("gray50", "red", "red", "blue", "blue"),
pch = c(19, 1, NA, NA, NA),
lty = c(NA, NA, 2, 1, 3),
lwd = c(NA, NA, 2, 2, 2)
)
The red open points are the support vectors returned by the fitted model. For points outside the tube, a larger circle indicates a larger violation of the tube. Points strictly inside the tube have zero dual coefficients. We deliberately use a linear kernel here so that the tube and support vectors are easy to see. Its systematic deviation from the dashed true function motivates the nonlinear extension.
4 Penalized SVR with RKHS
The SVR formulation can be naturally extended to nonlinear regression problems using the RKHS framework. By the Representer Theorem, the penalized component has a finite representation. With \(d_i=\alpha_i-\alpha_i'\), write
\[ f(x) = b+\sum_{i=1}^nd_iK(x,x_i). \]
Let \(\bd=(d_1,\ldots,d_n)^\T\) and let \(\bK\) be the kernel matrix with entries \(K_{ij}=K(x_i,x_j)\). The corresponding finite-dimensional problem is
\[ \min_{\bd\in\RR^n,\ b\in\RR} \quad \frac{1}{n}\sum_{i=1}^n L_\epsilon\big(y_i,(\bK\bd)_i+b\big) + \lambda\bd^\T\bK\bd. \]
Here \(b\) is an unpenalized intercept. This is useful because not every RKHS contains constant functions; the linear and homogeneous polynomial kernels are examples. The penalty \(\lambda\bd^\T\bK\bd\) is the squared RKHS norm of the penalized component and controls its complexity.
This normalized objective uses the same \(1/n\) loss convention as kernel ridge regression. Comparing it with the standard \(C\)-form above gives
\[ \lambda=\frac{1}{2nC}, \qquad\text{or equivalently}\qquad C=\frac{1}{2n\lambda}. \]
The regularization parameter, the width \(\epsilon\) of the insensitive tube, and any kernel parameters, such as the Gaussian bandwidth, may be selected by cross-validation using prediction performance on validation data.
The \(\epsilon\)-insensitive loss is nondifferentiable when \(y_i-f(x_i)=\pm\epsilon\). The dual formulation is a quadratic program, and common implementations use decomposition or SMO-type algorithms. A direct primal implementation could instead use subgradient methods.
The important distinction from kernel ridge regression is the loss function. Squared loss uses every residual, while the \(\epsilon\)-insensitive loss ignores observations strictly inside the tube. The Representer Theorem still gives a finite kernel expansion, and the KKT conditions make it sparse because only support vectors have nonzero dual coefficients. See Smola and Schölkopf (2004) for a detailed treatment.