Reinforcement learning (RL) is a framework in which an agent learns how to choose actions by repeatedly interacting with an environment, using rewards as its guide.

Supervised learning uses training data consisting of pairs of inputs and their correct outputs. In reinforcement learning, by contrast, the action chosen by the agent changes the state of the environment, and a reward is given as a result. By accumulating this experience, the agent learns how to choose actions that yield high rewards over the long term.

What matters here is not only the reward obtained at a single point in time. Because the current action also affects future states and rewards, reinforcement learning is treated as sequential decision-making, which involves a flow of events over time.

1.4.1 Basic Structure of Reinforcement Learning

Reinforcement learning consists mainly of the following elements.

Table 1.4-1 Main components of reinforcement learning

Element	Role
Environment	The system with which the agent interacts
Agent	The entity that selects actions according to the state
State	The situation of the environment at a given time
Action	An operation the agent can choose to perform on the environment
Reward	A scalar value given in response to the outcome of an action
Policy	The rule for selecting actions according to the state

The agent selects an action $a_t$ based on the state $s_t$ at time $t$. In response to that action, the environment moves to the next state $s_{t+1}$ and returns a reward $r_{t+1}$ to the agent. By repeating this process, the agent learns the relationship between actions and their outcomes.

Choosing the same action in the same state does not always lead to the same next state. For this reason, reinforcement learning may treat changes in the environment as probabilistic.

Markov Decision Process

Reinforcement learning problems are generally expressed using a Markov decision process (MDP). An MDP consists of the following elements, as expressed in Eq. (1.4-1).

Here, $\mathcal{S}$ is the set of states, $\mathcal{A}$ is the set of actions, $P$ is the state transition probability, $R$ is the function that determines the reward, and $\gamma$ is the discount factor, which adjusts how much weight is placed on future rewards.

This representation organizes, within a single framework, how the next state and reward are determined when a given action is chosen in a given state.

1.4.2 Evaluating Actions, Including Future Rewards

In reinforcement learning, judging whether an action is good or bad by looking only at the immediate reward may fail to properly evaluate future outcomes. Therefore, the return, which combines the rewards obtained from time $t$ onward, is used.

In this article, the return is denoted by $G_t$. When future rewards are discounted and summed, the return is expressed by Eq. (1.4-2).

In Eq. (1.4-2), powers of $\gamma$ are used so that rewards further away in time receive smaller weights. The goal of reinforcement learning is generally to learn a policy that maximizes the expected value of this return.

1.4.3 Main Challenges in Reinforcement Learning

Temporal Credit Assignment Problem

The effect of an action does not always appear immediately as a reward. When a reward is obtained after passing through multiple states and actions, it is necessary to determine which actions contributed to that outcome, and to what extent.

The problem of correctly linking a reward obtained later to the actions chosen before it is called the temporal credit assignment problem.

Sparse Rewards

In some environments, the agent receives almost no reward even after choosing many actions, and a reward is given only when a specific condition is met. Such rewards are called sparse rewards.

When a long sequence of actions is required before any reward is obtained, it is difficult for the agent to find out which actions lead to desirable outcomes. Sparse rewards therefore make both exploration and temporal credit assignment more difficult.

Exploration–Exploitation Trade-off

The agent must not only choose actions that past experience has shown to yield high rewards, but also check whether even better actions exist.

Acting on knowledge already obtained is called exploitation, and choosing actions that have not yet been sufficiently tried in order to gain new information is called exploration. Leaning too heavily toward exploitation may prevent the agent from discovering better actions. Leaning too heavily toward exploration, on the other hand, prevents the agent from making full use of actions already known to be effective. The need to balance the two is called the exploration–exploitation trade-off.

1.4.4 Example: Energy Management in a School Building

Consider energy management in a school building, including its classrooms, within the reinforcement learning framework. Here, the control system that decides how the building’s equipment is operated serves as the agent, and the building together with its energy management system serves as the environment.

Table 1.4-2 Correspondence example in school building energy management

Element	Correspondence in school building energy management
Environment	The school building and its energy management system
Agent	The control system that decides how equipment is operated
State	Information on energy consumption and indoor conditions
Action	Operations that change how equipment such as air conditioning is run
Reward	An evaluation reflecting energy consumption, thermal comfort, and indoor air quality
Policy	The rule for choosing equipment operations according to the state

In this example, simply reducing energy consumption is not enough. The comfort of the people using the building and the indoor air quality must also be taken into account. The reward therefore reflects multiple objectives, so that the agent can learn actions that balance them.

1.4.5 Relationship Between Reinforcement Learning and Deep Learning

Reinforcement learning is a framework for learning actions, and it does not in itself require deep learning.

However, in problems where states and actions are complex and their combinations are difficult to handle individually, deep neural networks can be used to approximate the relationships needed to choose actions from states. Methods that combine reinforcement learning and deep learning in this way are called deep reinforcement learning.

This article has focused on the basic structure and challenges of reinforcement learning. How policies and action values are learned, and where deep neural networks are used, will be covered in detail in later articles.

Summary

In reinforcement learning, the agent chooses actions according to the state of the environment and learns a policy based on the rewards it receives as a result.

Actions affect not only the immediate reward but also future states and rewards. Reinforcement learning therefore uses the return to evaluate long-term outcomes and seeks a way of choosing actions that maximizes the expected return.

However, there are challenges specific to sequential decision-making, such as the temporal credit assignment problem, in which rewards arrive with a delay; sparse rewards, in which opportunities to obtain a reward are rare; and the exploration–exploitation trade-off.

Reinforcement learning works without deep learning, but it can be combined with deep neural networks when dealing with complex states and actions.

About This Article

References
・Simon J. D. Prince (author), Toshiaki Kurokawa (translator), Shinsō Gakushū o Rikai suru I: Kyōshi-ari Gakushū to Moderu no Kiso [Understanding Deep Learning I: Supervised Learning and Model Fundamentals] (Japanese edition)

・Simon J. D. Prince (author), Toshiaki Kurokawa (translator), Shinsō Gakushū o Rikai suru II: Kyōshi-nashi Gakushū, Kyōka Gakushū, Rinri [Understanding Deep Learning II: Unsupervised Learning, Reinforcement Learning, and Ethics] (Japanese edition)

・Lorenzo Canese, Gian Carlo Cardarilli, Luca Di Nunzio, Rocco Fazzolari, Daniele Giardino, Marco Re, Sergio Spanò, “Multi-Agent Reinforcement Learning: A Review of Challenges and Applications,” Applied Sciences, 2021

・Arthur Aubret, Laetitia Matignon, Salima Hassas, “An Information-Theoretic Perspective on Intrinsic Motivation in Reinforcement Learning: A Survey,” Entropy, 2023

・Yassine Chemingui, Adel Gastli, Omar Ellabban, “Reinforcement Learning-Based School Energy Management System,” Energies, 2020

(Last accessed: October 12, 2026)


*This article was compiled based on the author’s understanding, with reference to the literature listed above.