Reinforcement Learning

---

Topics and Objectives

• Markov decision processes, the Markov property, and discounted rewards
• Value functions, Q-functions, and Bellman equations
• Bellman operators, contraction mappings, value iteration, and policy iteration
• Sampled Bellman updates, model-based versus model-free learning, and on-policy versus off-policy learning
• Fitted Q-iteration with linear, nonlinear, and forest-based function approximation and policy evaluation
• Policy gradients, REINFORCE, baselines, and actor-critic methods
• Maximum-entropy objectives, soft actor-critic, and behavior regularization in offline reinforcement learning

---

Module Schedule

This module contains four instructional weeks, excluding Fall Break. During the first three weeks, we will cover the following lecture materials as time permits. The fourth instructional week, December 1 and 3, will be a Present and Challenge week.

The lecture on November 12 is canceled. The remaining lecture meetings will be November 3, 5, 10, 17, and 19. The notes below are organized by topic and should be read in this order; a note may extend across class meetings.

• [Markov Decision Process and Bellman Equation] [Source Code]
• [Bellman Operators and Dynamic Programming] [Source Code]
• [Fitted Q-Iteration and the CartPole Simulation] [Source Code]
• [Policy Gradient Methods] [Source Code]
• [Entropy Regularization and Soft Actor-Critic] [Source Code]

We begin by defining policy value and the Bellman equation, then use Bellman updates to evaluate and improve policies. Fitted Q-iteration replaces these model-based updates with regression on observed transitions. Policy gradients give a direct way to improve a parameterized policy, leading to actor-critic methods and entropy regularization. The final note also discusses offline reinforcement learning and behavior regularization.

The [Cart-Pole Simulator] [Source Code] provides the environment details for the fitted Q-iteration example.

---

Homework

• TBA

---

Presentation Session

General logistics and scoring are described on the Present and Challenge page.

All teams are required to present in this module. Each team will propose, implement, and evaluate its own practical solution to a reinforcement learning task, then present its findings. Each team may use up to 12 minutes for its presentation and up to 3-5 minutes for Q&A. The specific time allocation may be shorter depending on the number of teams presenting in the session.

The basic requirements are:

1. Introduce your project’s problem and learning objective. Describe the environment or data available for learning.
2. Explain your approach and how it connects to the literature, or lectures we had.
3. Present your investigation and findings. Discuss the evidence, assumptions, and limitations, including how the learned policy is evaluated.

---