Introduction to Reinforcement Learning

Overview of Chapter 1 concepts from RL: An Introduction

This post captures my takeaways from Chapter 1 of the book Reinforcement Learning: An Introduction by Sutton and Barto, the bible of RL.

How do we learn?

Humans begins their learning by interacting with environment. Understanding the connection between our sensorimotors and environment through actions produces a wealth of information about cause and effect. Reinforcement learning (RL) focuses on developing computational approach to learning from interactions.

Reinforcement Learning

Reinforcement learning is learning what to do, how to map situations to actions, maximize a numerical reward signal. One has to discover actions that leads to maximum rewards. The most interesting and challenging part is that an action not only dictates the immediate reward but has effect on subsequent rewards as well. This delayed reward, trial-and-error learning are core foundations of RL. Understanding difference between Problems and Solution methods is very crucial in RL.

Reinforcement Learning = Problem + Solutions to the Problem + Field that studies the problem and it's solution methods

Is it supervised or un-supervised learning?

Most people tend to associate RL with Supervised or Un-supervised learning which is fundamentally incorrect. RL is a different ML paradigm alongside Supervised, Un-supervised and other learning paradigms.

Extrapolation vs Exploitation

The trade-off between Exploration and Exploitation is one of the biggest challenges in RL. At its heart, the difference is a conflict between using what you know versus learning something new.

• Exploitation (Harvesting): This is acting on your current knowledge to get the best immediate result. You stick to the path you believe is best to ensure a guaranteed reward1.
• Exploration (Discovering): This is acting to gather new information. You take a risk and try a path you haven't taken before (or don't know well) in hopes that it might be better than your current best.

The Dilemma: You cannot do both at the same time. If you only exploit, you might get a decent reward but miss out on a "jackpot" you didn't know existed. If you only explore, you waste time on bad choices and lose out on the steady rewards you could have had.

The exploration-exploitation trade-off dilemma

Characteristics of RL

The "Whole Package" vs "Isolated Skills"

RL focuses on the "Big Picture" rather than just isolated pieces of a puzzle. Imagine building a robot to play soccer.

• Other approaches (The Isolated Parts):

  • Supervised Learning: might focus entirely on teaching the robot to recognize a ball. The robot becomes a genius at spotting balls but doesn't know why it needs to spot them or what to do with them.
  • Planning Theories: might focus entirely on drawing perfect strategy diagrams on a whiteboard. But this assumes the robot knows exactly how the physics of the grass works and ignores that, in a real game, opponents move unpredictably and you don't have time to stop and think for 10 minutes.

• Reinforcement Learning (The Whole Package): RL starts with the assumption that the robot is on the field playing the game.

  • It has Senses: It sees the ball.
  • It has Actions: It can kick or run.
  • It has a Goal: Score a goal.
  • It has Uncertainty: The grass is slippery, and the opponent is fast.

RL studies how to make the robot function as a complete agent that senses, acts, and achieves goals all at once, rather than just perfecting one tiny skill (like ball recognition) in a vacuum. Because of this, RL naturally connects different isolated parts:

• If the agent plans ahead, RL must explain how planning works alongside real-time decision making.
• If the agent uses supervised learning, RL explains why it needs it and what exactly it should learn.
• RL also addresses how the agent learns and improves its model of the environment over time.

Putting all together

RL involves interaction between an active decision-making agent and environment. The agent seeks to achieve goal despite uncertainity present in its environment to maximize the reward signal. The agents actions affects the future state of environment thereby affecting opportunities available to agents in future. Correct choice requires to consider indirect and delayed consequences of actions thus may require planning. Agent can use it's experience to improve it's performance over time. Most of the times effect of actions cannot be fully predicted, thus agent must always monitor its environment and react accordingly. Even though the agent has some existing knowledge in the beginning of the task, interaction with environment is essential for adjusting behavior to exploit features of the task.

Entire RL (Problems and Solutions) revolves around the above passage.

Elements of RL

1. The Agent (The "Learner" or "Doer")

The Agent is the entity doing the learning and making the decisions. It is the active participant in the system.

• Role: Its job is to figure out the best actions to take to get the most reward.
• Capabilities: It can perceive the situation (see the State) and make a move (take an Action).

2. The Environment (The "World")

The Environment is everything outside the agent. It is the world in which the agent lives and operates.

• Role: It presents the problem to the agent. It responds to the agent's actions by changing the situation and giving feedback.
• Capabilities: It provides the State (current situation) and the Reward (feedback). The agent cannot control the environment directly; it can only influence it by taking actions.

3. The Policy (The "Brain" or Strategy)

The Policy is the agent's rulebook. It determines how the agent behaves at any given moment.

• What it does: It looks at the current situation (State) and decides what to do (Action)._
• Key Point: The policy alone is enough to determine how the agent acts. It can be a simple lookup table or a complex calculation.

4. The Reward Signal (The "Immediate Feedback")

The Reward Signal defines the immediate goal. It is a single number sent by the environment to tell the agent if things are going well or badly right now.

• What it does: It acts like a pleasure/pain sensor. The agent's only goal is to maximize the total amount of this reward over time.
• Key Point: Rewards change the Policy. If an action leads to a low reward (pain), the agent changes its policy to avoid doing that again.

5. The Value Function (The "Long-Term Vision")

While the Reward Signal tells you what is good now, the Value Function tells you what is good in the long run.

• What it does: It predicts the total amount of reward the agent can expect to get in the future starting from the current spot. It is "farsighted."
• Key Point: We make decisions based on Value, not just immediate Reward. We sometimes choose a difficult path (low immediate reward) because we know it leads to a better destination (high value).

6. Model of the Environment (The "Simulator")

This element is optional. A Model mimics the environment and allows the agent to predict what will happen before it actually acts.

• What it does: It allows for planning. The agent can "think" about a move without actually moving.
• Key Point: Not all RL systems have this. Some just learn by trial and error (Model-free), while others plan ahead (Model-based).
• Methods that use models for solving RL problems are called model-based methods while the methods that uses trial and error learning are called model-free methods.

Core elements of a Reinforcement Learning system

History of RL

The book covers the history in great depth, so I attempted to distill the most important ideas and contributions into the Mermaid diagram below. Each contribution is categorized into a different historical thread:

🔵 Blue: The Trial-and-Error Thread

• Represents: Psychology and Early AI.
• Meaning: This lineage focuses on "learning by doing." It traces the history of animal learning psychology (Thorndike, Pavlov) and early attempts to build machines that could learn from rewards and punishments without a teacher.

🔴 Red: The Optimal Control Thread

• Represents: Engineering, Mathematics, and Operations Research.
• Meaning: This lineage focuses on solving complex control problems using rigorous math (Calculus, Dynamic Programming). Historically, this was done "offline" using perfect models of the world, rather than by learning from experience.

🟢 Green: The Temporal-Difference (TD) Thread

• Represents: The "Bridge" between Psychology and Math.
• Meaning: This is a smaller but crucial thread. It focuses on learning by comparing predictions made at different times (e.g., "I predicted I would win 5 minutes ago, but now I predict I will lose"). This mechanism eventually allowed the math of Optimal Control to be applied to the learning of Trial-and-Error.

🟣 Purple: The Confluence (Merger)

• Represents: Modern Reinforcement Learning.
• Meaning: These boxes show where the independent threads finally merged. For example, Q-Learning (1989) is purple because it successfully combined the math of Optimal Control (Red) with the learning style of Temporal Difference (Green).

⚪ Grey (Dotted Border): The Confusion Period

• Represents: A specific era in the 1960s.
• Meaning: During this time, progress slowed because researchers often confused Reinforcement Learning with Supervised Learning (Pattern Recognition). This box represents a divergence where the field lost its focus on "trial-and-error" before finding it again.

Historical threads leading to modern Reinforcement Learning

References

1. Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction.

Tools

• Image Generation: Gemini, ChatGPT
• Editor: Antigravity
• History Diagram: Mermaid