Reinforcement Learning

---

Topics and Objectives

• Markov decision processes, the Markov property, and discounted rewards
• Value functions, Q-functions, and Bellman equations
• Bellman operators, contraction mappings, and value iteration
• Policy evaluation, policy improvement, and model-based versus model-free learning
• Fitted Q-iteration with linear, nonlinear, and forest-based function approximation
• Policy gradients, occupancy measures, and on-policy versus off-policy learning
• Maximum-entropy objectives, soft actor-critic, and behavior regularization in offline reinforcement learning

---

Module Schedule

This module contains four instructional weeks, excluding Fall Break. During the first three weeks, we will cover the following lecture materials as time permits. The fourth instructional week, December 1 and 3, will be a Present and Challenge week.

• [Markov Decision Process and Bellman Equation] [Source Code]
• [Bellman Equation and Contraction Mapping] [Source Code]
• [Fitted Q-Iteration and the CartPole Simulation] [Source Code]
• [Policy Gradient Methods] [Source Code]
• [Policy Iteration and Soft Actor-Critic] [Source Code]

The fitted Q-iteration lecture connects regression and tree ensembles to sequential decision making. The final lecture also discusses offline reinforcement learning and behavior regularization.

The [Cart-Pole Simulator] [Source Code] provides the environment details for the fitted Q-iteration example.

---

Homework

• TBA

---

Presentation Session

General logistics and scoring are described on the Present and Challenge page.

All teams are required to present in this module. Each team will propose and present its own project in reinforcement learning. Each team may use up to 12 minutes for its presentation and up to 3-5 minutes for Q&A. The specific time allocation may be shorter depending on the number of teams presenting in the session.

The basic requirements are:

1. Introduce your project’s problem and learning objective. Describe the environment or data available for learning.
2. Explain your approach and how it connects to the literature, or lectures we had.
3. Present your investigation and findings. Discuss the evidence, assumptions, and limitations, including how the learned policy is evaluated.

---