Reinforcement learning algorithms with mathematical derivations and reproductions of figures from Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed., 2020).
Broader MDP theory lives in Sequential-Decision-Making; the Sutton-notation foundations are also included here as derivations 01–03. Deep-RL methods live in Deep-Reinforcement-Learning.
git clone https://github.com/SaiSampathKedari/Reinforcement-Learning
cd Reinforcement-Learning
python -m venv .venv && source .venv/bin/activate
pip install -e .
# or, with uv
uv syncRequires Python ≥ 3.10.
src/rl/: algorithms and environments (see the package map).notebooks/: experiments, one folder per environment (see the notebook index).mathematical_derivations/: derivations, proofs, and convergence analysis (see the derivation index).
| Category | Algorithms | Status |
|---|---|---|
| Model-free prediction | MC prediction, TD(0), n-step TD, TD(λ) | ● |
| Model-free control | Sarsa, Q-learning, n-step Sarsa, Sarsa(λ) | ● |
| Dynamic programming | policy evaluation; policy & value iteration | ◐ |
| Model-based & planning | Dyna-Q, prioritized sweeping | ○ |
| Value-function approximation | gradient MC, state aggregation; semi-gradient TD / Sarsa, n-step semi-gradient TD / Sarsa, tile coding | ◐ |
| Policy gradient | REINFORCE, REINFORCE with baseline | ○ |
| Actor-critic | one-step actor-critic, advantage actor-critic (A2C) | ○ |
| Trust-region & proximal methods | natural policy gradient, TRPO, PPO | ○ |
● implemented · ◐ partial · ○ planned
Self-contained PDF write-ups in a shared reading sequence with Deep RL (see the derivation index). This repository covers 01–20, with 12 reserved for on-policy control with approximation; Deep RL shares 13–20 and continues from 21.
- MDP & dynamic-programming foundations (01–03) — policy evaluation in Sutton notation, Bellman operators, policy improvement theorem.
- Tabular & approximation — MC control (ES), ε-soft MC control, TD & eligibility traces, Sarsa(0), n-step Sarsa, Sarsa(λ), Q-learning, value-function approximation.
- Policy gradient & actor-critic — policy gradient theorem, average-reward PG, trajectory route, PG preliminaries, REINFORCE, actor-critic, baseline & advantage, GAE.
| Environment | Source | Status | Notebook |
|---|---|---|---|
| Blackjack | S&B §5.1 | ● | 01_blackjack |
| Windy Gridworld | S&B Example 6.5 | ● | 02_windy_gridworld |
| Random Walk | S&B Examples 6.2 / 7.1 / 9.1 | ● | 03_random_walk |
| Cliff Walking | S&B Example 6.6 | ○ | |
| FrozenLake | Gymnasium | ○ | |
| Dyna Maze | S&B Example 8.1 | ○ | |
| Mountain Car | S&B Example 10.1 | ● | 04_mountain_car |
| Acrobot | Classic control | ○ | |
| CartPole | Classic control | ○ | |
| Short Corridor | S&B Example 13.1 | ○ |
● implemented · ○ planned
A sequence from mathematical foundations to deep RL:
- Foundations — Real Analysis · Probability & Distribution Theory · Statistical Inference Theory
- RL theory — Sequential Decision Making
- This repo — Reinforcement Learning: S&B algorithms and figure reproductions
- Next — Deep Reinforcement Learning
- Email: sampath@umich.edu
- LinkedIn: sai-sampath-kedari
Blackjack: optimal policy π* and value V* (S&B Fig 5.2)
|
Windy Gridworld: value V(s) and 15-step optimal path (S&B Ex. 6.5)
|
Random Walk: gradient MC with state aggregation vs true vπ (S&B Fig 9.1)
|
Mountain Car: cost-to-go −maxa q̂(s,a,w) learned by semi-gradient Sarsa (S&B Fig 10.1)
|
Mountain Car: the learned greedy policy in action — reverse up the left slope to build momentum, then accelerate through to the goal
|
|