Although the spline approach in the previous lecture was successful and came with nice theoretical guarantees, it cannot be easily extended to high-dimensional data. If we use the tensor product construction, the number of basis functions will quickly explode with the scale of \(m^p\), where \(m\) is the number of basis functions in each dimension and \(p\) is the number of dimensions. This is known as the curse of dimensionality. But the theoretical results we saw for smoothing splines indicate that, within that particular Sobolev space, the minimizer has a finite representation determined by the observed data points. This suggests that we may be able to avoid explicitly constructing all \(m^p\) tensor-product basis functions. It does not remove the statistical curse of dimensionality, and that particular result may not be easily extended to other situations, as we need to carefully choose the space of functions to work with.
Can we have a more general framework in which a chosen kernel defines a rich space of functions and the data still lead to finite computation? The Reproducing Kernel Hilbert Space (RKHS) provides one such framework, with many nice properties. Hence the goal of this lecture is to properly define an RKHS. The key result here is the Moore–Aronszajn theorem, which establishes a one-to-one correspondence between positive-definite kernel functions and RKHSs. We will also show several examples that could help us understand the construction of an RKHS.
The development of this lecture is technical, and it is important for us to understand the details. The statistical payoff is concrete: the kernel determines the function space, the RKHS norm measures function complexity, and the reproducing property links point evaluation to an inner product. Kernel ridge regression combines these ingredients into an estimator. The Representer Theorem gives conditions under which a regularized optimization problem over an RKHS has a finite-dimensional solution determined by the observed data. The smoothing-spline result provides a concrete prototype for this general principle.
A Hilbert space is a complete inner product space. An inner product space that is not necessarily complete is sometimes called a pre-Hilbert space. A typical example is the Euclidean space \(\RR^n\) with the standard dot product as the inner product. But when we use a Hilbert space, we are often interested in an infinite-dimensional setting, such as a space of functions \(f: \RR^p \to \RR\). Each value \(f(x)\) is a scalar, but representing the entire function may require infinitely many coordinates or basis functions.
An inner product space equips these functions with a geometric structure, such as distance or angle, described by the inner product \(\langle \cdot, \cdot \rangle\). For our purpose, we will only deal with real-valued functions. Hence the inner product is a map
\[ \langle \cdot, \cdot \rangle: \cH_0 \times \cH_0 \to \RR, \]
where \(\cH_0\) is some space of functions. Besides containing a set of functions, we should have the following properties of the inner product for \(\cH_0\): For any \(f, g, h \in \cH_0\) and \(a, b \in \RR\), we have
The distance between two functions \(f\) and \(g\) in this space can be defined as
\[ d(f, g) = \| f - g \| = \sqrt{\langle f - g, f - g \rangle}. \]
The last step to make this space a Hilbert space is to ensure that it is complete. Completeness means that any Cauchy sequence in this space converges to a limit that is also in this space. A sequence \(\{f_n\}\) in \(\cH_0\) is called a Cauchy sequence if for any \(\epsilon > 0\), there exists an integer \(N\) such that for all \(m, n > N\), we have
\[ \| f_n - f_m \| < \epsilon, \]
using the norm we defined previously. Completing the pre-Hilbert space \(\cH_0\) means adding the limits of its Cauchy sequences, or intuitively, “filling all the holes” in the space. The original space \(\cH_0\) is dense in the completed space, which we denote as \(\cH = \overline{\cH_0}\).
An example of the Hilbert space is the space of square-integrable functions on the interval \([0, 1]\), denoted as \(L_2[0, 1]\). This space is defined as
\[ L_2[0, 1] = \left\{ f: [0, 1] \rightarrow \RR \quad \text{such that} \int_0^1 f(x)^2 d x < \infty \right\}, \]
where we require that the function is square-integrable. Strictly, \(L_2[0,1]\) identifies functions that agree almost everywhere, so its elements are equivalence classes rather than pointwise-defined functions. The inner product on this space is defined as
\[ \langle f, g \rangle_{L_2} = \int_0^1 f(x) g(x) d x, \]
which gives us a norm on the space:
\[ \| f \|_{L_2} = \sqrt{\langle f, f \rangle_{L_2}}. \]
This is a nice space, but it may have some undesirable properties for function estimation. For example, when two functions are different only on a measure-zero set, they are considered the same in this space. Therefore, the evaluation functional \(L_x(f) = f(x)\) is not even well-defined on \(L_2\) equivalence classes. Even if we restrict attention to continuous representatives, point evaluation is not continuous under the \(L_2\) norm: a small change in the \(L_2\) norm may lead to a large change in the function value at a specific point \(x\).1 This is not desirable when we want to find candidates in this class of functions to fit the data.
The first question we may ask is what are these functions \(f\) in the space \(\cH\)? Let’s use the following idea to define some functions we might be interested in. First, we always deal with an input space (or domain) \(\cX\) from the data that you observe. Elements in this space could be a vector of features in a typical regression problem, but they can also be as complex as an image or a graph. The beauty of RKHS is that it allows us to work with these complex objects as long as we can define a proper kernel function \(K(\cdot, \cdot)\) on the input space. Note that this kernel function has two inputs, and each of them should be an element in the input space \(\cX\). In some sense, this kernel function can be thought of as a generalized similarity measure between two instances from the input space. We will only talk about real-valued functions, but the construction can be extended to complex-valued functions. Now, let’s take a look at the function obtained by fixing one of the inputs to be \(x_1\), a particularly chosen point in \(\cX\):
\[K(\cdot, x_1) : \cX \to \RR, \quad \text{for any fixed } x_1 \in \cX.\]
By fixing \(x_1\), \(K(\cdot, x_1)\) becomes a function of the first argument that maps an input observation (\(\in \cX\)) to a real number (\(\in \RR\)). Hence, this effectively defines a function in the space of functions from \(\cX\) to \(\RR\). The output could represent a similarity score between the input observation and \(x_1\). Now we have one such function, and we can generate many such functions by varying \(x_1\) in the input space \(\cX\), hence creating a space of functions.
If we consider many of such \(x_1\) points in \(\cX\), e.g., \(x_1, x_2, \ldots, x_n\), then we can construct a space of functions that are linear combinations of these kernel functions centered at \(x_1, \ldots, x_n\):
\[\cH_0 = \left\{ f: f(\cdot) = \sum_{i=1}^n \alpha_i K(\cdot, x_i), \quad \alpha_i \in \RR, x_i \in \cX, n \in \NN \right\}.\]
This space contains a rich set of functions. Each particular expansion uses \(n\) kernel sections centered at selected points \(x_1,\ldots,x_n \in \cX\). The space \(\cH_0\) contains all such finite expansions and is generally not \(n\)-dimensional; the kernel sections also need not be linearly independent. As the number and locations of the centers vary, their combinations can be quite flexible. Let’s look at an example of this in the two-dimensional space \(\cX\):
# pick a grid of points to evaluate the function
ngrid = 100
grid.points <- seq(-2, 2, length.out = ngrid)
# define a Gaussian kernel function
gaussian.kernel <- function(x, y, sigma = 0.4) {
exp(-sum((x - y)^2) / (2 * sigma^2))
}
# define a function that calculates the kernel function of a given center
getKernel <- function(center, grid.points, sigma = 0.4) {
ngrid <- length(grid.points)
K.values <- matrix(0, nrow = ngrid, ncol = ngrid)
for (i in 1:ngrid) {
for (j in 1:ngrid) {
point <- c(grid.points[i], grid.points[j])
K.values[i, j] <- gaussian.kernel(point, center, sigma)
}
}
return(K.values)
}
# calculate the kernel function centered at several different points
K1 <- getKernel(center = c(0.2, 0.3), grid.points)
K2 <- getKernel(center = c(-0.5, -0.4), grid.points)
K3 <- getKernel(center = c(1, -1), grid.points)
K4 <- getKernel(center = c(-0.6, 0.3), grid.points)
K5 <- getKernel(center = c(0.4, -0.5), grid.points)
K6 <- getKernel(center = c(-0.3, -1.3), grid.points)
# define a sum of the kernel functions
K.sum <- 1.2*K1 + 1.3*K2 + K3 + 1.2*K4 + 1.6*K5 - 1.3*K6
# plot the kernel function
# define transparent colors
opscale = list(
c(0, 0), # z=0 → fully transparent
c(0.5, 0.5), # midway → half transparent
c(1, 1) # max → opaque
)
library(plotly)
plot_ly(x = grid.points, y = grid.points, z = K1, type = "surface",
opacityscale = opscale, showscale = FALSE) %>%
layout(paper_bgcolor='transparent') %>%
add_surface(z = K2, showscale = FALSE, opacityscale = opscale) %>%
add_surface(z = K3, showscale = FALSE, opacityscale = opscale) %>%
add_surface(z = K4, showscale = FALSE, opacityscale = opscale) %>%
add_surface(z = K5, showscale = FALSE, opacityscale = opscale) %>%
add_surface(z = K6, showscale = FALSE, opacityscale = opscale) %>%
add_surface(z = K.sum, showscale = FALSE,
surfacecolor = matrix(1, nrow = nrow(K.sum), ncol = ncol(K.sum)),
colorscale = list(c(0, "darkred"), c(1, "darkred")),
opacity = 0.7) %>%
layout(title = "Gaussian Kernel Functions",
scene = list(xaxis = list(title = "x1"),
yaxis = list(title = "x2"),
zaxis = list(title = "Kernel Value", range = c(-1.5, 2.5))))The plot above shows six Gaussian kernel functions centered at different points in the two-dimensional space, and their linear combination (in red). To be specific, the red surface is defined as
\[\begin{align} f(\cdot) =& 1.2 K(\cdot, (0.2, 0.3)) + 1.3 K(\cdot, (-0.5, -0.4)) + K(\cdot, (1, -1)) \\ &+ 1.2 K(\cdot, (-0.6, 0.3)) + 1.6 K(\cdot, (0.4, -0.5)) - 1.3 K(\cdot, (-0.3, -1.3)), \end{align}\]
where \(K(\cdot, \cdot)\) is a Gaussian kernel function. As you can see, by combining these kernel functions, we can create a variety of shapes, which all live in this space of functions. And the more kernel sections we have, the more flexible the combination could be.
Although we have defined a space of functions, it is not yet a pre-Hilbert space without properly defining the geometric structure, i.e., the inner product. The inner product allows us to measure the lengths, angles and orthogonality of functions in this space. Working towards our RKHS, we will define the inner product in a way that is closely related to the kernel function itself. For a symmetric kernel and any two functions \(f(\cdot) = \sum_{i=1}^n \alpha_i K(\cdot, x_i)\) and \(g(\cdot) = \sum_{j=1}^m \beta_j K(\cdot, y_j)\), both in \(\cH_0\), we define the inner product as:
\[\begin{align} \langle f, g \rangle_{\cH_0} &= \left\langle \sum_{i=1}^n \alpha_i K(\cdot, x_i), \sum_{j=1}^m \beta_j K(\cdot, y_j) \right\rangle_{\cH_0} \\ &\doteq \sum_{i=1}^n \sum_{j=1}^m \alpha_i \beta_j K(x_i, y_j). \end{align}\]
This definition does not depend on the particular finite expansions. Indeed,
\[ \sum_{i=1}^n \sum_{j=1}^m \alpha_i \beta_j K(x_i,y_j) = \sum_{j=1}^m \beta_j f(y_j) = \sum_{i=1}^n \alpha_i g(x_i). \]
The middle expression is independent of the representation of \(f\), and the last expression is independent of the representation of \(g\).
For example, if we look at the kernel sections used at the beginning of this development, we have
\[ \langle K(\cdot, x_i), K(\cdot, x_j) \rangle_{\cH_0} = K(x_i, x_j). \]
The map \(\Phi_K(x)=K(\cdot,x)\) is called the canonical feature map. The identity above gives
\[ K(x,x')=\langle\Phi_K(x),\Phi_K(x')\rangle_{\cH_0}. \]
It might be quite puzzling at this point why we use the kernel function itself to define the inner product. If you are interested in some alternative choices, you may look at a space we have used previously in the spline example, the second-order Sobolev space \(\cW_2^2\) on the interval \([0, 1]\)2 or simply the \(L_2\) space. But the choice we made here is crucial for making the space an RKHS, as we will see later. Let’s first make sure that this is a valid inner product by checking our previous required properties.
It now remains to check the positive-definiteness property. Here, we would require our choice of the kernel function to satisfy the following property:
Definition (positive-definite kernel)
A symmetric kernel function \(K: \cX \times \cX \to \RR\) is called a positive-definite kernel if for any \(n \in \NN\), any points \(x_1, x_2, \ldots, x_n \in \cX\), and any coefficients \(\alpha_1, \alpha_2, \ldots, \alpha_n \in \RR\), we have
\[ \sum_{i=1}^n \sum_{j=1}^n \alpha_i \alpha_j K(x_i, x_j) \geq 0. \]
Under this convention, every kernel matrix is positive semidefinite. Some authors reserve the term positive-definite for the stronger condition in which the inequality is strict for every nonzero coefficient vector and distinct input points.
With this property, the proposed bilinear form is nonnegative. For any \(f(\cdot)=\sum_{i=1}^n\alpha_iK(\cdot,x_i) \in \cH_0\),
\[\begin{align} \langle f, f \rangle_{\cH_0} &= \balpha^T \bK \balpha \geq 0. \end{align}\]
where \(\balpha = (\alpha_1, \ldots, \alpha_n)^T\) and \(\bK\) is the kernel matrix with \(K_{ij} = K(x_i, x_j)\). Notice that \(\balpha^T\bK\balpha=0\) does not necessarily imply \(\balpha=0\), since the kernel sections may be linearly dependent.
What we need to show is that zero norm implies the zero function. If \(\langle f,f\rangle_{\cH_0}=0\), then for any \(x\in\cX\) and \(t\in\RR\), positive-definiteness gives
\[ 0 \leq \langle f+tK(\cdot,x),f+tK(\cdot,x)\rangle_{\cH_0} =2t f(x)+t^2K(x,x). \]
Since this holds for every \(t\), we must have \(f(x)=0\). Conversely, if \(f\) is the zero function, then
\[ \langle f,f\rangle_{\cH_0} =\sum_{i=1}^n\alpha_i f(x_i)=0. \]
Hence this is a valid inner product on \(\cH_0\), making it a pre-Hilbert space.
An RKHS is a Hilbert space of functions in which every point-evaluation functional \(L_x(f)=f(x)\) is continuous. We now complete \(\cH_0\) and then verify that the resulting space has this property. To make \(\cH_0\) a Hilbert space, we ensure that it is complete, i.e., any Cauchy sequence in this space converges to a limit that is also in this space. We denote the completed space as
\[ \cH = \overline{\cH_0} = \overline{\text{span}}\{ K(\cdot, x) : x \in \cX \} \]
In the following, we will show why this completed space can be identified with an RKHS.
We first establish the key property that makes the completed space an RKHS: the reproducing property. This property states that for any function \(f \in \cH\) and any point \(x \in \cX\), we have
\[ f(x) = \langle f, K(\cdot, x) \rangle_{\cH}. \]
This means that the value of the function \(f\) at any point \(x\) can be obtained by taking the inner product of \(f\) with the kernel function centered at \(x\). Hence evaluation can be “reproduced” by the inner product. Let’s look at two things: why this is true and why is this important.
Let’s take an example \(f(\cdot) = \sum_{i=1}^n \alpha_i K(\cdot, x_i) \in \cH_0\). Evaluating this function at \(x\) gives
\[\begin{align} f(x) &= \sum_{i=1}^n \alpha_i K(x, x_i) = \sum_{i=1}^n \alpha_i K(x_i, x) \quad \text{(by symmetry)} \\ &= \sum_{i=1}^n \alpha_i \big\langle K(\cdot, x_i), K(\cdot, x) \big\rangle_{\cH_0} \quad \text{(by definition of inner product)} \\ &= \left\langle \sum_{i=1}^n \alpha_i K(\cdot, x_i), K(\cdot, x) \right\rangle_{\cH_0} \quad \text{(by linearity of inner product)} \\ &= \big\langle f, K(\cdot, x) \big\rangle_{\cH_0}. \end{align}\]
This shows that the reproducing property holds for any function in the pre-Hilbert space \(\cH_0\). It also shows that point evaluation is bounded on \(\cH_0\):
\[ |f(x)|=\big|\langle f,K(\cdot,x)\rangle_{\cH_0}\big| \leq \|f\|_{\cH_0}\sqrt{K(x,x)}. \]
Therefore, evaluation at \(x\) extends uniquely and continuously to the completion \(\cH\). If \(f_n\in\cH_0\) and \(f_n\to f\) in \(\cH\), we define \(f(x)=\lim_{n\to\infty}f_n(x)\). The bound above ensures that this limit is independent of the approximating sequence. By continuity of the inner product,
\[ f(x)=\lim_{n\to\infty}f_n(x)=\lim_{n\to\infty}\langle f_n,K(\cdot,x)\rangle_{\cH}=\langle f,K(\cdot,x)\rangle_{\cH}. \]
Thus the completion can be identified with a Hilbert space of functions, and the reproducing property holds throughout \(\cH\). In particular, every point-evaluation functional is continuous, so \(\cH\) is an RKHS.
The reproducing property gives a useful stability bound. For any function \(f \in \cH\), we can look at its evaluation at any two points \(x, x' \in \cX\):
\[\begin{align} |f(x) - f(x')| &= \big| \big\langle f, K(\cdot, x) \big\rangle_{\cH} - \big\langle f, K(\cdot, x') \big\rangle_{\cH} \big| \quad \text{(by reproducing property)} \\ &= \big| \big\langle f, K(\cdot, x) - K(\cdot, x') \big\rangle_{\cH} \big| \quad \text{(by linearity of inner product)} \\ &\leq \big\| f \big\|_{\cH} \cdot \big\| K(\cdot, x) - K(\cdot, x') \big\|_{\cH} \quad \text{(by Cauchy-Schwarz inequality)} \\ &= \big\| f \big\|_{\cH} \cdot d_K(x, x'), \end{align}\]
where \(d_K(x,x')=\sqrt{K(x,x)+K(x',x')-2K(x,x')}\) is the kernel-induced pseudometric.3 Thus every \(f\in\cH\) is Lipschitz with respect to \(d_K\), with Lipschitz constant at most \(\|f\|_{\cH}\). This inequality does not by itself imply smoothness in the original input coordinates. For that conclusion, we also need \(d_K(x,x')\to0\) whenever \(x'\to x\), as happens for continuous kernels such as the Gaussian kernel. Under this condition, nearby inputs have nearby function values, and a smaller RKHS norm gives tighter control of their variation. Hence, the norm \(\|f\|_{\cH}\) can be used as a measure of the complexity of the function, and more commonly \(\|f\|_{\cH}^2\) can be used as a penalty in a regularized optimization problem. This will be demonstrated in the next lecture.
The RKHS we defined above is very special, as it has a one-to-one correspondence with the choice of the kernel function. This is established by the Moore–Aronszajn theorem (Aronszajn 1950). Sometimes it was stated with only one direction, but here we will state it in both directions.
Theorem (Moore–Aronszajn) For any symmetric positive-definite kernel function \(K: \cX \times \cX \to \RR\), there exists a unique RKHS \(\cH\) of functions \(f: \cX \to \RR\) with reproducing kernel \(K\). And conversely, for any RKHS \(\cH\) of functions \(f: \cX \to \RR\), there exists a unique symmetric positive-definite kernel function \(K: \cX \times \cX \to \RR\) that is the reproducing kernel of \(\cH\).
“\(\Longrightarrow\)” The forward direction is commonly referred to as the Moore–Aronszajn theorem. Existence follows from the construction above. To show uniqueness, suppose that two RKHSs \(\cH_1\) and \(\cH_2\) have the same reproducing kernel \(K\). In either space, \(\langle K(\cdot,x),K(\cdot,x')\rangle=K(x,x')\), so the inner products agree on \(\cH_0=\operatorname{span}\{K(\cdot,x):x\in\cX\}\). Moreover, \(\cH_0\) is dense in either RKHS: if \(f\) is orthogonal to every \(K(\cdot,x)\), then the reproducing property gives \(f(x)=0\) for every \(x\), so \(f\) is the zero function. Therefore, both \(\cH_1\) and \(\cH_2\) are completions of the same pre-Hilbert space with the same inner product, and hence they coincide.
“\(\Longleftarrow\)” Conversely, suppose that \(\cH\) is an RKHS. Since the evaluation functional \(L_x(f)=f(x)\) is continuous, the Riesz representation theorem gives a unique \(k_x\in\cH\) such that \(f(x)=\langle f,k_x\rangle_{\cH}\) for every \(f\in\cH\). Define \(K(x,x')=k_{x'}(x)=\langle k_{x'},k_x\rangle_{\cH}\). Then \(K\) is symmetric, \(K(\cdot,x)=k_x\), and
\[ \sum_{i=1}^n\sum_{j=1}^n\alpha_i\alpha_jK(x_i,x_j) =\left\|\sum_{i=1}^n\alpha_i k_{x_i}\right\|_{\cH}^2\geq0. \]
Thus \(K\) is positive-definite and has the reproducing property by construction. It is unique because \(k_x\) is the unique Riesz representative of \(L_x\) for every \(x\).
For \(x,x'\in\RR^p\), some common kernel functions are:
But let’s look at some other interesting examples. The Brownian motion kernel defined on the interval \([0, 1]\):
\[ K(x, x') = \min(x, x'), \quad x, x' \in [0, 1]. \]
The RKHS associated with this kernel is the first-order Sobolev space starting from 0 (Berlinet and Thomas-Agnan 2011), defined as
\[ \cH = \left\{ f: [0, 1] \rightarrow \RR \quad \text{such that } f \text{ is absolutely continuous},\ f(0)=0,\ f'\in L_2[0,1] \right\}, \]
with the inner product defined as
\[ \langle f, g \rangle_\cH = \int_0^1 f'(x) g'(x) d x = \langle f', g' \rangle_{L_2}. \]
We can check some simple properties from what we have learned. A kernel section in this space is \(K(\cdot,t)=\min(\cdot,t)\), which is a piecewise linear function with a knot at \(t\). Its derivative is the step function \(1_{[0,t]}\). The inner product between two such kernel sections is the kernel function:
\[\begin{align} \langle K(\cdot, x_1), K(\cdot, x_2) \rangle_\cH = \int_0^1 1_{[0, x_1]}(u) \cdot 1_{[0, x_2]}(u) d u = \min(x_1, x_2) = K(x_1, x_2). \end{align}\]
The reproducing property also holds:
\[\begin{align} \langle f, K(\cdot, x) \rangle_\cH &= \int_0^1 f'(u) \frac{\partial}{\partial u} K(u, x) d u \\ &= \int_0^1 f'(u) 1_{[0, x]}(u) d u \\ &= \int_0^x f'(u) d u = f(x) - f(0) = f(x). \end{align}\]
Not every symmetric function proposed as a kernel is positive-definite. For example, consider the hyperbolic tangent kernel (Ong et al. 2004):
\[ K(x, x') = \tanh(\alpha x^T x' + \beta), \]
where \(\alpha>0\), \(\beta<0\), and \(\tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}}\). For some parameter choices and data sets, its kernel matrix can have both positive and negative eigenvalues. In this case, an RKHS is not the appropriate framework. A reproducing kernel Krein space provides a more general framework. When the indefinite kernel admits a decomposition \(K=K_+-K_-\), where both \(K_+\) and \(K_-\) are positive-definite, any function in the space can be decomposed as
\[ f = f_+ + f_-, \]
where \(f_+\in\cH_+\) and \(f_-\in\cH_-\) belong to the RKHSs associated with \(K_+\) and \(K_-\), respectively. The Krein-space inner product has the form \([f,g]=\langle f_+,g_+\rangle_{\cH_+}-\langle f_-,g_-\rangle_{\cH_-}\).
In practice, we may want to define new kernels for our specific problems. There are several ways to do this:
A positive-definite kernel determines both an RKHS and the geometry of its functions through the reproducing property. The RKHS norm provides a natural measure of function complexity. In the next lecture, kernel ridge regression will combine squared-error loss with this norm penalty and compute the estimator through a kernel matrix.
In the Hilbert space \(L^2([0,1])\), the evaluation functional \(L_x(f)=f(x)\) is not necessarily continuous. For example, define a sequence of functions \[ f_n(t) = \begin{cases} n^{1/3}(1 - nt), & 0 \le t \le \tfrac{1}{n},\\ 0, & \tfrac{1}{n} < t \le 1. \end{cases} \] All such functions are continuous. And \(\|f_n - 0\|_{L^2}^2 \leq \int_0^{1/n} n^{2/3} \, dt = n^{-1/3} \to 0\), so \(f_n \to 0\) in \(L^2\). But at the point \(t=0\), \(f_n(0)=n^{1/3} \to \infty\). Thus, convergence in the \(L^2\)-norm does not imply pointwise stability of function values. This shows why bounded point-evaluation functionals in an RKHS are useful.↩︎
which is the sum of \(L_2\) inner products of the functions and their first two derivatives, with \[ \langle f, g \rangle_{L_2} = \int_0^1 f(x) g(x) d x \] And this also gives us a norm on the space: \[ \| f \|_{\cW_2^2} = \sqrt{\langle f, f \rangle_{\cW_2^2}}. \]↩︎
For the Gaussian kernel, \(d_K(x,x')=\sqrt{2-2\exp\{-\|x-x'\|^2/(2\sigma^2)\}}\), which approaches zero as \(\|x-x'\|\to0\) and is bounded above by \(\sqrt{2}\).↩︎