In our previous lectures, we developed the foundations of reinforcement learning through the Markov Decision Process framework, the Bellman equation, and the contraction properties that justify dynamic programming. We introduced several algorithms such as fitted Q-iteration and policy gradient. These two perspectives highlight different sides of the same problem: one learns (action) values and extracts a greedy policy, while the other directly updates a parameterized policy, possibly using an estimated value function.
The Bellman operators and dynamic programming lecture introduced policy iteration, which alternates between evaluating a policy and greedily improving it. The policy-gradient lecture instead updated a parameterized policy using returns or a critic. However, a greedy policy commits to an action, and in continuous action spaces computing an exact maximizer of \(Q(s,a)\) can be difficult. When the critic is estimated, this maximizer can also be sensitive to estimation errors. We now ask how policy improvement changes when the objective rewards randomness in the policy.
These limitations motivate the introduction of Soft Actor-Critic (SAC, Haarnoja et al. (2018)), which approximates a soft, entropy-regularized form of policy iteration. The entropy term yields a Boltzmann-type improvement, and a parameterized actor makes sampling and optimization practical for continuous actions. SAC reuses past transitions through off-policy updates, usually while continuing to collect new data. Learning reliably from a fixed offline dataset requires additional attention to distribution shift and value estimation errors.
Before introducing Soft Actor Critic, we first review the idea of using entropy in a reinforcement learning objective. In the standard framework, the goal is to maximize the expected cumulative reward. In the maximum entropy formulation, we instead consider an augmented objective that adds an entropy term to encourage the policy to remain stochastic. Formally, the objective
\[ J(\pi) = \mathbb{E}_{s_0\sim\mu_0}[V^{\pi}(s_0)] \]
is replaced with the discounted maximum-entropy return
\[ J(\pi) =\E_{\pi} \Big[ \sum_{t=0}^{\infty} \gamma^t \big( r(s_t,a_t) + \alpha\,\cH(\pi(\cdot|s_t)) \big) \Big], \]
where \(\mu_0\) is a fixed initial state distribution, \(\alpha>0\) controls the entropy reward, and, for a finite action space,
\[ \cH(\pi(\cdot|s)) = -\sum_{a}\pi(a|s)\log\pi(a|s) \]
Note that here the entropy term is not a function of the actually observed action \(a_t\), but rather a function of the policy distribution at state \(s_t\), represented by the \(\cdot\) notation. Here, if we write it into the expectation \(\E_\pi\), we should have
\[ -\sum_a \pi(a|s)\log\pi(a|s) = -\E_{a\sim\pi(\cdot|s)} [ \log\pi(a|s) ]. \]
Substituting this into the maximum-entropy return gives the equivalent expected objective
\[ J(\pi) =\E_{\pi} \Big[ \sum_{t=0}^{\infty} \gamma^t \big( r(s_t,a_t) - \alpha \log\pi(a_t|s_t) \big) \Big], \]
The two per-step expressions are equal after averaging over the action, not for each realized action:
\[ \E_{a_t\sim\pi(\cdot|s_t)}[r(s_t,a_t)] + \alpha\mathcal{H}(\pi(\cdot|s_t)) = \E_{a_t\sim\pi(\cdot|s_t)}\big[r(s_t,a_t) - \alpha\log\pi(a_t|s_t)\big]. \]
For the rest of this lecture, \(V^\pi\) and \(Q^\pi\) denote entropy-regularized, or “soft,” values. They differ from the ordinary reward-only values in the preceding lectures. Under this objective, the value function becomes
\[ V^\pi(s) = \E_{a\sim\pi(\cdot|s)} \big[ Q^\pi(s,a) - \alpha \log \pi(a|s) \big], \]
and the corresponding soft Q-function is defined as
\[ Q^\pi(s,a) = r(s,a) + \gamma\,\E_{s'\sim P(\cdot|s,a)}\left[ V^\pi(s') \right]. \]
Under this convention, \(Q^\pi(s,a)\) includes entropy rewards starting at the next state; the current state’s entropy enters \(V^\pi(s)\) through the action expectation. These definitions therefore reflect the discounted maximum-entropy objective above. In addition, the optimal soft value function satisfies the soft Bellman optimality equation:
\[ \begin{aligned} V^\ast(s) =& \max_{\pi(\cdot|s)} \; \E_{a\sim\pi(\cdot|s)} \big[ Q^\ast(s,a) - \alpha \log \pi(a|s) \big] \\ =& \max_{\pi(\cdot|s)} \;\Big\{\E_{a\sim\pi(\cdot|s)} \big[ Q^\ast(s,a) \big] + \alpha\,\cH(\pi(\cdot|s))\Big\}. \end{aligned} \]
A key consequence of this formulation is that the optimal policy at state \(s\) admits a closed-form solution of Boltzmann type, as we will derive next.
We now derive the policy improvement at a particular state \(s\) for a fixed soft Q-function \(Q(s,a)\). For finite actions and \(\alpha>0\), choose a distribution \(\pi(\cdot|s)\) that maximizes
\[ \begin{aligned} \max_{\pi(\cdot|s)} & \quad \E_{a\sim\pi(\cdot|s)} \big[ Q(s,a) \big] \;+\; \alpha\,\cH(\pi(\cdot|s)) \\[2mm] \text{subject to} & \quad \sum_{a\in\cA}\pi(a|s)=1,\qquad \pi(a|s)\geq0. \end{aligned} \]
To solve this optimization problem, we write the Lagrangian
\[ \cL(\pi,\lambda) = \sum_{a}\pi(a) Q(s,a) - \alpha\sum_{a}\pi(a)\log\pi(a) - \lambda\Big(\sum_{a}\pi(a)-1\Big), \]
where \(\lambda\) is the Lagrange multiplier that enforces the normalization constraint. With finite Q-values and \(\alpha>0\), the maximizer is interior and the objective is strictly concave. Taking the derivative with respect to \(\pi(a)\) and setting it to zero therefore yields the unique maximizer:
\[ Q(s,a) - \alpha(1+\log\pi(a)) - \lambda = 0. \]
Solving for \(\log\pi(a)\),
\[ \log\pi(a) = \frac{1}{\alpha} Q(s,a) - \Big(\frac{\lambda}{\alpha} + 1\Big). \]
Exponentiating both sides gives
\[ \pi(a) = \exp\!\Big(\frac{1}{\alpha}Q(s,a)\Big) \cdot \exp\!\Big(-\frac{\lambda}{\alpha}-1\Big). \]
The second factor is a constant that does not depend on \(a\) and serves as a normalization factor. Enforcing \(\sum_a \pi(a|s)=1\) gives
\[ \exp\!\Big(-\frac{\lambda}{\alpha}-1\Big) = \frac{1}{Z(s)}, \qquad Z(s) = \sum_{a\in\cA}\exp\!\Big(\frac{1}{\alpha}Q(s,a)\Big). \]
Thus, the maximizing policy for the given \(Q\) is
\[ \pi_Q(a|s) = \frac{\exp\!\big(\frac{1}{\alpha} Q(s,a)\big)}{Z(s)}. \]
This has the same form as the softmax policy parameterization from the policy-gradient lecture, but the roles differ. There, free parameters determine the policy’s scores and are learned by a gradient update. Here, the scores are the fixed critic values divided by \(\alpha\), and the Boltzmann rule solves the entropy-regularized improvement problem for that critic.
This is the optimal soft policy when \(Q=Q^\ast\); with a current policy’s soft Q-function, it is an improvement step. As \(\alpha\) approaches zero, the distribution concentrates on actions that maximize \(Q(s,a)\), recovering greedy improvement. Larger \(\alpha\) gives more weight to entropy. Adding the same constant to all Q-values at a state leaves the normalized policy unchanged, but evaluating its probabilities still requires normalization. The KL objective below avoids computing \(Z(s)\) because it is constant with respect to the actor parameters.
For continuous actions, replace sums by integrals and use differential entropy. The Boltzmann density requires \(0<Z(s)=\int_{\cA}\exp(Q(s,a)/\alpha)\,da<\infty\). For example, bounded Q-values on a bounded action domain of positive volume satisfy this condition; bounded Q-values on all of \(\mathbb R^m\) do not.
The SAC algorithm approximates soft policy iteration. In continuous action spaces, normalizing and sampling directly from \(\exp(Q(s,a)/\alpha)\) can be difficult. Instead, we use a tractable policy class \(\Pi=\{\pi_\phi\}\), such as transformed Gaussian policies on a bounded action domain, and fit the actor to the Boltzmann distribution induced by the current Q-function. This fit optimizes an improvement objective for the current critic; it does not by itself find the policy with globally optimal return.
However, the next question is how do we evaluate what is “close” when we parameterize our policy? Here we use the KL divergence from the actor to the Boltzmann distribution. Exact soft policy improvement minimizes this divergence separately at each state. With parameters shared across states, SAC instead minimizes its average over states sampled from the replay buffer \(\cD\):
\[ \pi_{\phi}^\text{new} = \arg\min_{\pi_\phi \in \Pi} \E_{s\sim\cD}\Bigg[ \text{KL}\!\left( \pi_\phi(\cdot|s) \;\Vert\; \frac{ \exp(Q(s,\cdot)/\alpha) }{ Z(s) } \right)\Bigg]. \]
Recall that the KL divergence between two distributions \(P\) and \(Q\) is defined as
\[ \begin{aligned} \text{KL}(P\|Q) &= \int P(x)\log\frac{P(x)}{Q(x)}\,dx \\ &= \E_{x\sim P} \big[\log P(x) - \log Q(x)\big]. \end{aligned} \]
Hence, for our case, the KL divergence gives
\[ \text{KL}(\pi_\phi(\cdot|s) \| \pi_Q(\cdot|s)) = \E_{a\sim\pi_\phi(\cdot|s)} \bigg[ \log\pi_\phi(a|s) - \frac{1}{\alpha}Q(s,a) + \log Z(s) \bigg]. \]
Since \(\log Z(s)\) is constant with respect to \(\phi\), it does not affect the minimizer. Hence, the policy improvement step can be written as
\[ \pi_{\phi}^\text{new} = \arg\min_{\pi_\phi \in \Pi} \E_{s\sim\cD,\,a\sim\pi_\phi(\cdot|s)} \big[ \log\pi_\phi(a|s) - \frac{1}{\alpha}Q(s,a) \big]. \]
Gradient descent approximately minimizes this actor objective while holding the critic fixed; sampled actions must still depend on \(\phi\), for example through reparameterized sampling. The critic is updated using a soft Bellman target. This resembles fitted Q-iteration as a regression step, but its next action is drawn from the current actor rather than chosen by a hard maximum. For a simplified single-critic presentation, parameterize \(Q\) by \(\theta\) and minimize
\[ \cL_Q(\theta) = \E_{(s,a,r,s')\sim\cD} \big[ \big( Q_\theta(s,a) - y \big)^2 \big]. \]
where the target \(y\) is given by the soft Bellman equation
\[ y = r + \gamma\,\E_{a'\sim\pi_\phi(\cdot|s')} \big[ Q_{\bar\theta}(s',a') - \alpha \log \pi_\phi(a'|s') \big], \]
where \(\bar\theta\) denotes a slowly updated target copy of the critic parameters, held fixed when differentiating the critic loss. The next action is sampled from the current actor. For a transition that truly terminates the episode, the target is \(y=r\), with no continuation term. Standard SAC implementations also use two critics and the smaller target Q-value to reduce overestimation.
SAC alternates approximate versions of these two steps during training; the exact policy-iteration convergence guarantee does not automatically apply to neural network updates:
Recall the data distinctions introduced earlier. Classical dynamic programming assumes a known model and does not require on-policy samples. When learning from data, on-policy evaluation uses data generated by the policy being evaluated, whereas off-policy evaluation uses data from another behavior policy \(\mu\). For example, we could collect driving data by following the current policy and use those data to evaluate it. In a medical treatment problem, however, collecting new data under each proposed policy may be costly or impractical. We may instead need to evaluate a new treatment policy using data collected under an existing one.
Separately, online learning collects new interactions during training, whereas offline learning uses a fixed dataset. SAC is off-policy because it can reuse transitions from earlier policies in a replay buffer, even while training online. Applying it to a fixed offline dataset is possible, but off-policy updates alone do not prevent the actor from choosing actions whose values are poorly estimated.
The replay buffer supplies observed transitions for critic fitting and states for actor fitting. The actions inside the actor objective and the next actions inside the critic target are sampled from the current actor, not taken directly from the behavior dataset. With a fixed offline dataset, these actions can lie outside well-observed regions. The actor may then exploit critic errors, and bootstrapping can propagate those errors (Fujimoto et al. 2019). Coverage of the policies considered during training and accurate value estimates matter; off-policy learning alone does not make standard SAC a reliable offline method.
In practice, we may also regularize the policy to stay close to the behavior policy, rather than (or in addition to) maximizing entropy. This can limit distribution shift in offline RL. For example, a KL penalty can discourage deviation from a reference policy (Wu et al. 2019). Other offline methods restrict or compare actions using different mechanisms (Fujimoto et al. 2019; Kumar et al. 2019). Such penalties encourage proximity; by themselves, they do not guarantee safety or improved return.