Previously we discussed methods that estimate the value or the Q functions, and then derive a policy by taking the argmax over the action space. An alternative approach is to directly parameterize the policy \(\pi\), and optimize the policy parameters to maximize the expected cumulative return. This is known as the policy-based method. In this lecture, we will derive the policy gradient theorem, which provides an explicit expression for the gradient of the expected return with respect to the policy parameters. We will then use sampled returns and estimated values to construct REINFORCE and actor-critic updates. The longer proof and occupancy calculations are collected in the optional section after these algorithms.
Recall that we can have a trajectory \(\bh_t\) that contains all the information of \(s_t, a_t\) up to time \(t\):
\[ \bh_t = (s_0, a_0, \dots, s_t, a_t, s_{t+1}). \]
Here we will ignore the reward for simplicity (think of them as deterministic functions of states and actions if you like). The density function of a trajectory \(\bh_t\) under a policy \(\pi\) can be written as
\[ p_\pi(\bh_t) = \mu_0(s_0) \prod_{i=0}^{t} \pi(a_i \mid s_i) P(s_{i+1} \mid s_i, a_i), \]
where \(\mu_0\) is the initial state distribution, \(P\) is the transition probability, and \(\pi\) is the policy used to generate the data. Our goal is to find a good policy \(\pi\) that gives us high cumulative return.
In our previous lecture, we did not really specify any functional form of the policy \(\pi\). Instead, we estimated the Q-function, and used the argmax operation to derive a greedy policy. Here, we will directly parameterize the policy \(\pi\) as \(\pi_\theta(a \mid s)\), where \(\theta\) is the parameter vector. A common choice can be a softmax version:
\[ \pi_\theta(a \mid s) = \frac{\exp(\phi(s,a)^\top \theta)}{\sum_{a'} \exp(\phi(s,a')^\top \theta)}, \]
where \(\phi(s,a)\) is a feature vector for the state-action pair \((s,a)\). A simple example is the logistic regression for binary action space. With this parameterization in mind, we can consider the objective function for policy optimization as
\[ \begin{aligned} J(\theta) =& \, \E_{\, \bh \sim p_{\theta} \,} \bigg[ \sum_{t=0}^\infty \gamma^t r(s_t, a_t) \bigg] \\ =& \, \E_{s_0 \sim \mu_0} \Big[ V^{\pi_\theta}(s_0) \Big] \end{aligned} \]
Here \(\bh=(s_0,a_0,s_1,a_1,\dots)\) denotes the full trajectory. This is understood as the expected discounted reward when following the policy \(\pi_\theta\), but with the initial state \(s_0\) also integrated out according to the initial state distribution \(\mu_0\) that represents a population that we are interested in. Here the notation \(V^{\pi_\theta}(s)\) is the value function under policy \(\pi_\theta\). Our eventual goal is to find the optimal parameter \(\theta^*\) that maximizes \(J(\theta)\), but as an intermediate step, we may be just concerned about finding a new policy that improves upon the current policy. In general, we would treat this as an optimization problem that maximizes \(J(\theta)\) by updating \(\theta\) along the gradient direction. This would involve computing the gradient \(\nabla_\theta J(\theta)\). This general approach is known as the policy gradient method.
The following theorem gives us an explicit expression for the policy gradient. For this derivation, assume finite state and action spaces, bounded rewards, \(0<\gamma<1\), and a continuously differentiable policy with \(\pi_\theta(a\mid s)>0\). The initial distribution, rewards, and transition probabilities do not depend on \(\theta\). These conditions justify differentiation and the limiting argument in the optional proof below.
(Policy Gradient Theorem) Given the objective function \(J(\theta)\), the gradient can be expressed as \[ \nabla_\theta J(\theta) = \frac{1}{1-\gamma} \, \mathbb{E}_{(s,a)\sim d^{\pi_\theta}} \Big[ \nabla_\theta \log \pi_\theta(a\mid s)\, Q^{\pi_\theta}(s,a) \Big]. \] Here, \(d^{\pi_\theta}(s,a)\) is the normalized discounted state-action visitation distribution under policy \(\pi_\theta\), defined as (for simplicity in tabular case): \[ d^{\pi_\theta}(s,a) = (1-\gamma) \sum_{t=0}^\infty \gamma^t \, \mathbb{P}_{\pi_\theta}(s_t = s, a_t = a). \]
The distribution \(d^{\pi_\theta}\) describes discounted visits starting from \(\mu_0\), and generally differs from a stationary distribution. Its state marginal is \(d^{\pi_\theta}(s)=\sum_a d^{\pi_\theta}(s,a)\), with \(d^{\pi_\theta}(s,a)=d^{\pi_\theta}(s)\pi_\theta(a\mid s)\). Equivalently, the theorem says
\[ \nabla_\theta J(\theta) =\E_{\pi_\theta}\left[ \sum_{t=0}^{\infty}\gamma^t Q^{\pi_\theta}(s_t,a_t)\nabla_\theta\log\pi_\theta(a_t\mid s_t) \right]. \]
Thus, we do not need to differentiate the Q-function when computing the policy gradient. We need its value under the current policy and the derivative of the log policy, also called the score. The factor \(1/(1-\gamma)\) appears when using the normalized distribution \(d^{\pi_\theta}\); the trajectory expression instead carries the weights \(\gamma^t\).
The policy gradient theorem provides a general mechanism for updating a parameterized policy toward higher long-term return. The most direct estimator uses trajectories from the current policy \(\pi_\theta\). First, suppose we could estimate the Q-function for every action. An all-action version of the gradient would use
\[ \nabla_\theta J(\theta) =\frac{1}{1-\gamma}\E_{s\sim d^{\pi_\theta}}\left[ \sum_a Q^{\pi_\theta}(s,a)\nabla_\theta\pi_\theta(a\mid s) \right]. \]
However, the Q-function still needs to be estimated, and summing or integrating over all actions may be inconvenient. The classical REINFORCE algorithm (Williams 1992) instead samples an action and uses its subsequent return to estimate \(Q^{\pi_\theta}(s_t,a_t)\).
For an episode with actions at times \(0,\ldots,T-1\) and terminal state \(s_T\), write the observed reward as \(R_t\). The return from time \(t\) is
\[ G_t=\sum_{k=t}^{T-1}\gamma^{k-t}R_k, \qquad \E_{\pi_\theta}[G_t\mid s_t,a_t]=Q^{\pi_\theta}(s_t,a_t). \]
Here the episode ends at a true terminal state, after which rewards are zero. For a continuing task, cutting a trajectory at an arbitrary time omits the tail return; the finite sum is then a truncated approximation. Notice that \(G_t\) discounts rewards relative to time \(t\). The outer weight \(\gamma^t\) is still needed for the objective \(J(\theta)\) defined above.
The algorithm works as follows:
Using the full episodic returns gives an unbiased gradient estimate under the stated conditions, but a single trajectory can give a noisy update. A batch of independent episodes can be averaged before updating the policy.
We return to the three-state model from the Bellman and policy-iteration lecture, with the same transition matrices, rewards, and \(\gamma=0.7\). To make the gradient calculation visible, we first use exact policy evaluation rather than sampled returns. Let
\[ p_s=\pi_\theta(1\mid s)=\frac{\exp(\theta_s)}{1+\exp(\theta_s)}, \qquad \pi_\theta(2\mid s)=1-p_s. \]
This is a tabular special case of the earlier softmax parameterization, with one parameter for each state. Its score at a visited state is \(\partial\log\pi_\theta(a\mid s)/\partial\theta_s=\mathbf{1}\{a=1\}-p_s\). Summing over the two actions in the theorem gives
\[ \frac{\partial J(\theta)}{\partial\theta_s} =\frac{d^{\pi_\theta}(s)}{1-\gamma} p_s(1-p_s)\left[Q^{\pi_\theta}(s,1)-Q^{\pi_\theta}(s,2)\right]. \]
Start from \(\theta=(0,0,0)^\top\), so both actions have probability \(1/2\), and take \(\mu_0=(1,0,0)\). The state occupancy is \(d^{\pi_\theta}=(1-\gamma)\mu_0(I-\gamma P^{\pi_\theta})^{-1}\); its matrix derivation is given in the optional section.
P1 = matrix(c(0.8, 0.1, 0.1,
0.05, 0.05, 0.9,
0.2, 0.2, 0.6), byrow=TRUE, nrow=3)
P2 = matrix(c(0.5, 0.25, 0.25,
0.1, 0.8, 0.1,
0.8, 0.1, 0.1), byrow=TRUE, nrow=3)
r = matrix(c(5, 3,
2, 2.5,
3, 2), byrow=TRUE, nrow=3)
gamma = 0.7
mu0 = c(1, 0, 0)
theta = rep(0, 3)
p = plogis(theta)
P_pi = p * P1 + (1 - p) * P2
V = solve(diag(3) - gamma * P_pi, p * r[, 1] + (1 - p) * r[, 2])
Q = cbind(r[, 1] + gamma * P1 %*% V,
r[, 2] + gamma * P2 %*% V)
d = drop((1 - gamma) * mu0 %*% solve(diag(3) - gamma * P_pi))
gradient = d / (1 - gamma) * p * (1 - p) * (Q[, 1] - Q[, 2])
eta = 0.5
theta_new = theta + eta * gradient
knitr::kable(data.frame(State = 1:3, Q1 = Q[, 1], Q2 = Q[, 2],
Occupancy = d, Gradient = gradient,
Prob1_before = p, Prob1_after = plogis(theta_new)),
digits = 4)| State | Q1 | Q2 | Occupancy | Gradient | Prob1_before | Prob1_after |
|---|---|---|---|---|---|---|
| 1 | 12.8259 | 10.3811 | 0.6781 | 1.3816 | 0.5 | 0.6661 |
| 2 | 8.9877 | 9.0625 | 0.1447 | -0.0090 | 0.5 | 0.4989 |
| 3 | 10.0651 | 9.8259 | 0.1771 | 0.0353 | 0.5 | 0.5044 |
The update increases the probability of action 1 most strongly in state 1, where its action value is substantially larger. This is one local gradient step, and the Q-values evaluate the current randomized policy. They are not the optimal Q-values from value iteration. REINFORCE estimates the same gradient by replacing the exact action values with returns and averaging the discounted score contributions over trajectories.
An arbitrary state-dependent baseline \(b(s)\) can be subtracted from the return without changing the policy gradient. This is similar to the mean function adjustment in outcome weighted learning, as long as \(b(s)\) is not a function of the action. For each state,
\[ \begin{aligned} \E_{a\sim\pi_\theta(\cdot\mid s)}\left[ b(s)\nabla_\theta\log\pi_\theta(a\mid s) \right] &=b(s)\sum_a\nabla_\theta\pi_\theta(a\mid s)\\ &=b(s)\nabla_\theta 1=0. \end{aligned} \]
Consequently, the gradient can be written as
\[ \nabla_\theta J(\theta) =\frac{1}{1-\gamma}\E_{(s,a)\sim d^{\pi_\theta}}\left[ \big\{Q^{\pi_\theta}(s,a)-b(s)\big\} \nabla_\theta\log\pi_\theta(a\mid s) \right]. \]
The corresponding REINFORCE update replaces \(G_t\) by \(G_t-b(s_t)\). A useful baseline can reduce the variance of the estimate; an arbitrary baseline need not do so. The baseline is held fixed when forming the actor gradient.
One idea is to take \(b(s)=V^{\pi_\theta}(s)\). The difference
\[ A^{\pi_\theta}(s,a)=Q^{\pi_\theta}(s,a)-V^{\pi_\theta}(s) \]
is called the advantage function. It measures how much better an action is than the average action under the current policy at that state. A score update encourages the sampled action when its estimated advantage is positive and discourages it when the estimate is negative. With parameters shared across states, it can also change other action probabilities.
REINFORCE waits for the future rewards to estimate the action value. Can we instead use a fitted value function and one observed transition? The Bellman equation gives
\[ A^{\pi_\theta}(s_t,a_t) =\E_{\pi_\theta}\left[ R_t+\gamma V^{\pi_\theta}(s_{t+1})-V^{\pi_\theta}(s_t) \mid s_t,a_t\right]. \]
Here \(V^{\pi_\theta}(s)=0\) at a true terminal state. Thus the sampled temporal-difference (TD) error is an unbiased estimate of the advantage conditional on \((s_t,a_t)\) when the true value function is used. In practice, a critic estimates \(V^{\pi_\theta}\) by a function \(V_w\), with parameter \(w\). If \(D_t=1\) indicates true termination, its estimated TD error is
\[ \delta_t=R_t+\gamma(1-D_t)V_w(s_{t+1})-V_w(s_t). \]
Using this estimated advantage, basic actor and critic updates along an on-policy trajectory are
\[ \begin{aligned} \theta &\leftarrow\theta+\eta_\pi\gamma^t\delta_t \nabla_\theta\log\pi_\theta(a_t\mid s_t),\\ w &\leftarrow w+\eta_V\delta_t\nabla_w V_w(s_t). \end{aligned} \]
The actor changes the policy using the estimated advantage. The critic learns to evaluate the current policy, using the TD target \(R_t+\gamma(1-D_t)V_w(s_{t+1})\), held fixed when taking its gradient. This is the same sampled-backup idea used earlier, but the next value evaluates the current policy rather than taking a maximum over actions. The actor treats \(\delta_t\) as a fixed signal when taking the score gradient. The outer weight \(\gamma^t\) matches the discounted objective used here.
Compared with a full return, the TD error uses less future randomness, but it can be biased when the critic is inaccurate. Actor-critic methods alternate approximate evaluation and improvement, connecting the earlier policy-iteration idea to a parameterized policy. Approximate critics and finite step sizes do not give the exact policy-improvement guarantee.
A stochastic policy lets us explore different actions, but maximizing expected reward alone may lead the actor to concentrate its probability on a few actions early in training. In the next lecture, we will add a reward for the entropy of the policy. This changes the objective and leads to soft policy evaluation and soft actor-critic (Haarnoja et al. 2018). For a fixed critic, its Boltzmann improvement rule solves an entropy-regularized optimization problem; the softmax formula introduced here is simply a way to parameterize a policy.
We have used \(\theta\) for the actor parameters and \(\phi(s,a)\) for a feature vector. In the SAC lecture, the actor parameters are called \(\phi\) and the critic parameters are called \(\theta\). Their roles will be stated there; the feature vector \(\phi(s,a)\) is not used in that notation.
To prove this theorem, we start with the value function for a given state \(s\). Following the definition of the value function, under policy \(\pi_\theta\), we have
\[ V^{\pi_\theta}(s) = \sum_{a} \pi_\theta(a\mid s)\,Q^{\pi_\theta}(s,a). \]
Noticing that both \(\pi_\theta\) and \(Q^{\pi_\theta}\) depend on \(\theta\), we will differentiate \(V^{\pi_\theta}(s)\) with respect to \(\theta\) using the product rule:
\[ \nabla_\theta V^{\pi_\theta}(s) = \sum_{a} \big[ \nabla_\theta \pi_\theta(a\mid s)\big]\,Q^{\pi_\theta}(s,a) + \sum_{a} \pi_\theta(a\mid s)\,\nabla_\theta Q^{\pi_\theta}(s,a). \tag{1} \]
We cannot simply take the derivative of \(Q^{\pi_\theta}(s,a)\) directly, since it is a quantity that obeys its own Bellman equation. Thus we will express the Q function further into its Bellman equation form:
\[ Q^{\pi_\theta}(s,a) = r(s,a) + \gamma\,\mathbb{E}_{s' \sim P(\cdot\mid s,a)}\big[ V^{\pi_\theta}(s') \big]. \]
The reward function \(r\) and transition kernel \(P\) do not depend on \(\theta\) once the state and action are fixed. We have
\[ \nabla_\theta Q^{\pi_\theta}(s,a) = \gamma\,\mathbb{E}_{s' \sim P(\cdot\mid s,a)}\big[ \nabla_\theta V^{\pi_\theta}(s') \big]. \]
Substitute into the previous expression for \(\nabla_\theta V^{\pi_\theta}(s)\):
\[ \begin{aligned} \nabla_\theta V^{\pi_\theta}(s) &= \sum_{a} \big[ \nabla_\theta \pi_\theta(a\mid s)\big]\,Q^{\pi_\theta}(s,a) \\ &\quad + \gamma \sum_{a} \pi_\theta(a\mid s) \, \mathbb{E}_{s' \sim P(\cdot\mid s,a)}\big[ \nabla_\theta V^{\pi_\theta}(s') \big]. \end{aligned} \]
Now we will utilize a log-derivative trick. Noticing that
\[ \nabla_\theta \pi_\theta(a\mid s) = \pi_\theta(a\mid s)\,\nabla_\theta \log \pi_\theta(a\mid s). \]
This allows us to rewrite the first term in (1) as an expectation under the policy distribution \(\pi_\theta(\cdot\mid s)\):
\[ \begin{aligned} &\sum_{a} \big[ \nabla_\theta \pi_\theta(a\mid s)\big]\,Q^{\pi_\theta}(s,a) \\ =& \sum_{a} \pi_\theta(a\mid s)\,\nabla_\theta \log \pi_\theta(a\mid s)\,Q^{\pi_\theta}(s,a) \\ =& \, \mathbb{E}_{a \sim \pi_\theta(\cdot\mid s)}\big[ \nabla_\theta \log \pi_\theta(a\mid s)\,Q^{\pi_\theta}(s,a) \big]. \end{aligned} \]
For the second term in (1), observe that \[ \gamma \sum_{a} \pi_\theta(a\mid s) \mathbb{E}_{s' \sim P(\cdot\mid s,a)}\big[ \nabla_\theta V^{\pi_\theta}(s') \big] = \gamma\,\mathbb{E}_{a \sim \pi_\theta(\cdot\mid s),\,s' \sim P(\cdot\mid s,a)} \big[ \nabla_\theta V^{\pi_\theta}(s') \big]. \]
Thus we obtain the recursive relation \[ \nabla_\theta V^{\pi_\theta}(s) = \mathbb{E}_{a \sim \pi_\theta(\cdot\mid s)}\big[ \nabla_\theta \log \pi_\theta(a\mid s)\,Q^{\pi_\theta}(s,a) \big] + \gamma\,\mathbb{E}_{a \sim \pi_\theta(\cdot\mid s),\,s' \sim P(\cdot\mid s,a)}\big[ \nabla_\theta V^{\pi_\theta}(s') \big]. \tag{2} \]
The next step is to further unroll this recursion along the trajectory induced by the policy \(\pi_\theta\). We consider a more compact form of the derivative of the value function:
\[ \begin{aligned} &\nabla_\theta V^{\pi_\theta}(s_0)\\ =& \, \E_{a_0 \sim \pi_\theta} \big[ \nabla_\theta \log \pi_\theta(a_0\mid s_0) \,Q^{\pi_\theta}(s_0,a_0) + \gamma\,\E_{s_1 \sim P(\cdot\mid s_0,a_0)} \big[ \nabla_\theta V^{\pi_\theta}(s_1) \big] \mid s_0 \big] \\ =& \, \E_{a_0, s_1, a_1} \big[ \nabla_\theta \log \pi_\theta(a_0\mid s_0)\,Q^{\pi_\theta}(s_0,a_0) +\gamma\,\nabla_\theta \log \pi_\theta(a_1\mid s_1)\,Q^{\pi_\theta}(s_1,a_1) +\gamma^2\,\E_{s_2\sim P(\cdot\mid s_1,a_1)} \big[ \nabla_\theta V^{\pi_\theta}(s_2) \big] \mid s_0 \big] \\ =& \cdots \\ =& \,\E \Big[ \sum_{t=0}^{T} \gamma^t \nabla_\theta \log \pi_\theta(a_t\mid s_t)\,Q^{\pi_\theta}(s_t,a_t) +\gamma^{T+1} \nabla_\theta V^{\pi_\theta}(s_{T+1}) \mid s_0 \Big] \\ \xrightarrow{T\to\infty}& \,\E \Big[ \sum_{t=0}^{\infty} \gamma^t \nabla_\theta \log \pi_\theta(a_t\mid s_t)\,Q^{\pi_\theta}(s_t,a_t) | s_0 \Big]. \end{aligned} \]
The remainder vanishes because \(\nabla_\theta V^{\pi_\theta}\) is bounded in this finite setting and \(\gamma^{T+1}\to0\). By taking expectation over the initial state distribution \(\mu_0\), we have
\[ \begin{aligned} \nabla_\theta J(\theta) &= \E_{s_0 \sim \mu_0} \big[ \nabla_\theta V^{\pi_\theta}(s_0) \big] \\ &= \E \Big[ \sum_{t=0}^{\infty} \gamma^t \nabla_\theta \log \pi_\theta(a_t\mid s_t)\,Q^{\pi_\theta}(s_t,a_t) \Big]. \end{aligned} \]
in which the expectation is taken over the trajectory, including the initial state distribution, the policy, and the state transition dynamics. The form of this is still complicated since it involves the expectation over the infinite horizon. However, there is a neat trick to rewrite this expression using the discounted state-action visitation distribution.
We now introduce the normalized discounted occupancy measure, which describes how frequently each state is visited under a policy when future visits are geometrically discounted. This distribution plays a central role in expressing expected returns and analyzing policy optimization. It depends on both the policy and the initial state distribution, and is generally different from a stationary distribution. We will motivate this with a tabular case, but the idea extends to more general settings.
Let the initial state distribution \(\mu_0\) be written as a row vector in \(\mathbb{R}^{1 \times |\mathcal{S}|}\). Under a policy \(\pi\), the state distribution evolves as \[ \mu_{t+1}^\pi = \mu_t^\pi P^\pi, \] where \(P^\pi\) is the transition matrix induced by \(\pi\), with each row summing to one. Repeating this transition \(t\) times yields \[ \mu_t^\pi = \mu_0 (P^\pi)^t. \]
To reflect the discounted nature of future rewards, we weight future state distributions by \(\gamma^t\). The (unnormalized) discounted occupancy measure is defined as \[ d_\gamma^\pi = \sum_{t=0}^\infty \gamma^t \mu_t^\pi, \qquad 0 < \gamma < 1. \]
This is not a probability distribution; its total mass is \(1/(1-\gamma)\). Substituting the expression for \(\mu_t^\pi\), \[ \begin{aligned} d_\gamma^\pi &= \sum_{t=0}^\infty \gamma^t \mu_0 (P^\pi)^t \\ &= \mu_0 \sum_{t=0}^\infty (\gamma P^\pi)^t \\ &= \mu_0 (I - \gamma P^\pi)^{-1}. \end{aligned} \]
Since \(\| \gamma P^\pi \|_\infty = \gamma < 1\), the matrix inverse is well-defined. Multiplying by \(1 - \gamma\) normalizes the measure and yields the discounted state visitation distribution: \[ d^\pi = (1-\gamma)\, d_\gamma^\pi = (1-\gamma)\, \mu_0 (I - \gamma P^\pi)^{-1}. \]
Although it may be difficult to estimate this when the state space is large, we can establish a useful recursive relation for \(d^\pi\), which resembles the Bellman equation. Starting from the definition of \(d^\pi\), \[ d^\pi = (1-\gamma)\, \mu_0 + \gamma\, d^\pi P^\pi. \]
For a continuous state space with densities and a finite action space, the relation becomes \[ d^\pi(s) = (1-\gamma)\, \mu_0(s) + \gamma\, \int d^\pi(s') \sum_{a} \pi(a\mid s') P(s\mid s',a)\, ds'. \]
This structure mirrors the Bellman equation for value functions. One may also define the normalized discounted state-action occupancy: \[ d^\pi(s,a) = d^\pi(s)\,\pi(a\mid s), \]
which satisfies
\[ d^\pi(s,a) = (1-\gamma)\, \mu_0(s)\,\pi(a\mid s) + \gamma\, \int \sum_{a'} d^\pi(s',a')\, P(s\mid s',a')\,\pi(a\mid s)\, ds'. \]
For continuous actions with densities, replace the action sums by integrals. In practice, we may not have access to the transition dynamics \(P\) or the full state space. However, many modern methods estimate \(d^\pi\) (or density ratios relative to another policy) using only sampled trajectories by solving the fixed-point relation above. See Nachum et al. (2019) for an example.
To proceed, we first rewrite the trajectory-level expectation explicitly. For a bounded function \(f\), consider \[ \E_{\pi_\theta}\Bigg[ \sum_{t=0}^{\infty} \gamma^t\, f(s_t,a_t) \Bigg], \] where the expectation is taken over full trajectories \[ \bh = (s_0,a_0,s_1,a_1,\dots), \qquad a_t \sim \pi_\theta(\cdot\mid s_t),\; s_{t+1} \sim P(\cdot\mid s_t,a_t). \]
Writing this expectation explicitly, \[ \E_{\pi_\theta}\Bigg[ \sum_{t=0}^{\infty} \gamma^t\, f(s_t,a_t) \Bigg] = \int_{\bh} \Bigg( \sum_{t=0}^{\infty} \gamma^t\, f(s_t,a_t) \Bigg) p_{\pi_\theta}(d\bh), \] where \(p_{\pi_\theta}(d\bh)\) denotes the probability law of the full trajectory under policy \(\pi_\theta\). We then take the sum outside the integral. Below, \(\mu_t^{\pi_\theta}(s,a)\) denotes the joint state-action distribution at time \(t\); use sums in the finite setting and integrals when densities exist:
\[ \begin{aligned} =& \sum_{t=0}^{\infty} \gamma^t \int_{\bh} f(s_t,a_t)\, p_{\pi_\theta}(d\bh) \\ =& \sum_{t=0}^{\infty} \gamma^t \int_{s_t, \, a_t} f(s_t,a_t) \mu_t^{\pi_\theta}(s_t,a_t)\, da_t\, ds_t \\ =& \sum_{t=0}^{\infty} \gamma^t \int_{s, \, a} f(s,a) \mu_t^{\pi_\theta}(s,a)\, da\, ds \\ =& \int_{s, \, a} f(s,a) \Bigg( \sum_{t=0}^{\infty} \gamma^t \mu_t^{\pi_\theta}(s,a) \Bigg) da\, ds \\ =& \frac{1}{1-\gamma} \int_{s, \, a} f(s,a) d^{\pi_\theta}(s,a)\, da\, ds \\ =& \frac{1}{1-\gamma} \E_{(s,a)\sim d^{\pi_\theta}} \big[ f(s,a) \big]. \end{aligned} \]
Applying this result to our policy gradient expression, we set \[ f(s_t,a_t) = \nabla_\theta \log \pi_\theta(a_t\mid s_t)\, Q^{\pi_\theta}(s_t,a_t). \] This gives us the final form of the policy gradient theorem:
\[ \nabla_\theta J(\theta) = \frac{1}{1-\gamma} \, \mathbb{E}_{(s,a)\sim d^{\pi_\theta}} \Big[ \nabla_\theta \log \pi_\theta(a\mid s)\, Q^{\pi_\theta}(s,a) \Big]. \]
The policy gradient theorem provides a general mechanism for updating a parameterized policy toward higher long-term return. Its expectation is under the target policy’s discounted occupancy distribution. The most direct estimator uses trajectories from the current policy \(\pi_\theta\), retaining the discount weights in the trajectory expression above. Policy gradients can also be estimated from offline data, but then the available data come from a behavior policy \(\mu\). One approach reweights samples from \(d^\mu\) by an estimate of \(d^{\pi_\theta}(s,a)/d^\mu(s,a)\), together with an estimate of \(Q^{\pi_\theta}\). This requires \(d^\mu(s,a)>0\) wherever the target occupancy is positive; the state-action sampling distribution of the dataset must also match the denominator. Alternatively, one can use trajectory importance sampling, which can have high variance. The theorem alone does not remove these estimation and coverage problems.
In the online regime, classical algorithms such as REINFORCE (Williams 1992) and actor-critic methods apply the policy gradient theorem, but gradient estimates can have high variance and overly large updates can reduce performance. Trust Region Policy Optimization (TRPO, Schulman et al. (2015)) limits the KL divergence between old and new policies. Its theoretical motivation uses a monotonic-improvement bound, but the practical algorithm makes approximations and does not inherit an unconditional guarantee. Proximal Policy Optimization (PPO, Schulman et al. (2017)) instead clips the policy probability ratio inside a surrogate objective to discourage large updates. Clipping does not enforce a hard KL constraint or guarantee monotonic improvement. PPO has also been used in reinforcement learning from human feedback to optimize language models using rewards learned from human preferences.