Actor-Critic Algorithm in Reinforcement Learning

Last Updated : 22 Mar, 2024

Reinforcement learning (RL) stands as a pivotal component in the realm of artificial intelligence, enabling agents to learn optimal decision-making strategies through interaction with their environments.

Let’s Dive into the actor-critic algorithm, a key concept in reinforcement learning, and learn how it can improve your machine learning models.

Table of Content

What is the Actor-Critic Algorithm?
How Actor-Critic Algorithm works?
A2C (Advantage Actor-Critic)
Training Agent: Actor-Critic Algorithm
Advantages of Actor Critic Algorithm
Advantage Actor Critic (A2C) vs. Asynchronous Advantage Actor Critic (A3C)
Conclusion

What is the Actor-Critic Algorithm?

The actor-critic algorithm is a type of reinforcement learning algorithm that combines aspects of both policy-based methods (Actor) and value-based methods (Critic). This hybrid approach is designed to address the limitations of each method when used individually.

In the actor-critic framework, an agent (the “actor”) learns a policy to make decisions, and a value function (the “Critic”) evaluates the actions taken by the Actor.

Simultaneously, the critic evaluates these actions by estimating their value or quality. This dual role allows the method to strike a balance between exploration and exploitation, leveraging the strengths of both policy and value functions.

Key Components of Reinforcement Learning

Before delving into the actor-critic method, it’s crucial to understand the fundamental components of reinforcement learning (RL):

Agent: The entity making decisions and interacting with the environment.
Environment: The external system with which the agent interacts.
State: A representation of the current situation or configuration.
Action: The decision or move made by the agent.
Reward: The feedback received by the agent based on its actions.
Policy: The strategy or set of rules guiding the agent’s decision-making.

Roles of Actor and Critic

Actor: The actor makes decisions by selecting actions based on the current policy. Its responsibility lies in exploring the action space to maximize expected cumulative rewards. By continuously refining the policy, the actor adapts to the dynamic nature of the environment.
Critic: The critic evaluates the actions taken by the actor. It estimates the value or quality of these actions by providing feedback on their performance. The critic’s role is pivotal in guiding the actor towards actions that lead to higher expected returns, contributing to the overall improvement of the learning process.

Key Terms in Actor Critic Algorithm

There are two key terms:

Policy (Actor) :
- The policy, denoted as $\pi(a|s)$ , represents the probability of taking action a in state s.
- The actor seeks to maximize the expected return by optimizing this policy.
- The policy is modeled by the actor network, and its parameters are denoted by $\theta$
Value Function (Critic) :
- The value function, denoted as $V(s)$ , estimates the expected cumulative reward starting from state s.
- The value function is modeled by the critic network, and its parameters are denoted by w.

How Actor-Critic Algorithm works?

Actor Critic Algorithm Objective Function

The objective function for the Actor-Critic algorithm is a combination of the policy gradient (for the actor) and the value function (for the critic).
The overall objective function is typically expressed as the sum of two components:

Policy Gradient (Actor)

$\nabla_\theta J(\theta)\approx \frac{1}{N} \sum_{i=0}^{N} \nabla_\theta \log\pi_\theta (a_i|s_i)\cdot A(s_i,a_i)$

Here,

$J(θ)$ represents the expected return under the policy parameterized by $θ$
$π_\theta (a∣s)$ is the policy function
N is the number of sampled experiences.
$A(s,a)$ is the advantage function representing the advantage of taking action a in state s.
i represents the index of the sample

Value Function Update (Critic)

$\nabla_w J(w) \approx \frac{1}{N}\sum_{i=1}^{N} \nabla_w (V_{w}(s_i)- Q_{w}(s_i , a_i))^2$

Here,

$\nabla_w J(w)$ is the gradient of the loss function with respect to the critic’s parameters w.
N is number of samples
$V_w(s_i)$ is the critic’s estimate of value of state s with parameter w
$Q_w (s_i , a_i)$ is the critic’s estimate of the action-value of taking action a
i represents the index of the sample

Update Rules

The update rules for the actor and critic involve adjusting their respective parameters using gradient ascent (for the actor) and gradient descent (for the critic).

Actor Update

$\theta_{t+1}= \theta_t + \alpha \nabla_\theta J(\theta_t)$

Here,

$\alpha$ : learning rate for the actor
t is the time step within an episode

Critic Update

$w_{t} = w_t -\beta \nabla_w J(w_t)$

Here,

w represents the parameters of the critic network
$\beta$ is the learning rate for the critic

Advantage Function

The advantage function, $A(s,a)$ , measures the advantage of taking action a in state s over the expected value of the state under the current policy.

$A(s,a)=Q(s,a)−V(s)$

The advantage function, then, provides a measure of how much better or worse an action is compared to the average action.

These mathematical expressions highlight the essential computations involved in the Actor-Critic method. The actor is updated based on the policy gradient, encouraging actions with higher advantages, while the critic is updated to minimize the difference between the estimated value and the action-value.

A2C (Advantage Actor-Critic)

A2C (Advantage Actor-Critic) is a specific variant of the Actor-Critic algorithm that introduces the concept of the advantage function. This function measures how much better an action is compared to the average action in a given state. By incorporating this advantage information, A2C focuses the learning process on actions that have a significantly higher value than the typical action taken in that state.

While both leverage the actor-critic architecture, here’s a key distinction between them:

Learning from the Average: The base Actor-Critic method uses the difference between the actual reward and the estimated value (critic’s evaluation) to update the actor.
Learning from the Advantage: A2C leverages the advantage function, incorporating the difference between the action’s value and the average value of actions in that state. This additional information refines the learning process further.

Actor-Critic Algorithm Steps

The Actor-Critic algorithm combines these mathematical principles into a coherent learning framework. The algorithm involves:

Initialization:
- Initialize the policy parameters $\theta$ (actor) and the value function parameters $\phi$ (critic).
Interaction with the Environment:
- The agent interacts with the environment by taking actions according to the current policy and receiving observations and rewards in return.
Advantage Computation:
- Compute the advantage function A(s,a) based on the current policy and value estimates.
Policy and Value Updates:
- Simultaneously update the actor’s parameters $(\theta)$ using the policy gradient. The policy gradient is derived from the advantage function and guides the actor to increase the probabilities of actions that lead to higher advantages.
- Simultaneously update the critic’s parameters $(\phi)$ using a value-based method. This often involves minimizing the temporal difference (TD) error, which is the difference between the observed rewards and the predicted values.

The actor learns a policy, and the critic evaluates the actions taken by the actor. The actor is updated using the policy gradient, and the critic is updated using a value-based method. This combination allows for more stable and efficient learning in complex environments.

Training Agent: Actor-Critic Algorithm

Let’s understand how the Actor-Critic algorithm works in practice. Below is an implementation of a simple Actor-Critic algorithm using TensorFlow and OpenAI Gym to train an agent in the CartPole environment.

Import Libraries

Python

import numpy as np
import tensorflow as tf
import gym

2. Creating CartPole Environment

Create the CartPole environment using the gym.make() function from the Gym library because it provides a standardized and convenient way to interact with various reinforcement learning tasks.

Python

# Create the CartPole Environment
env = gym.make('CartPole-v1')

3. Defining Actor and Critic Networks

Actor and the Critic are implemented as neural networks using TensorFlow’s Keras API.
Actor network maps the state to a probability distribution over actions.
Critic network estimates the state’s value.

Python

# Define the actor and critic networks
actor = tf.keras.Sequential([
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(env.action_space.n, activation='softmax')
])

critic = tf.keras.Sequential([
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(1)
])

4. Defining Optimizers and Loss Functions

Adam optimizer is used for both the Actor and the Critic networks.

Python

# Define optimizer and loss functions
actor_optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
critic_optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)

5. Training Loop

Main training loop runs for a specified number of episodes (1000).
Agent interacts with the environment, and for each episode, it resets the environment and initializes the episode reward to 0.
The with tf.GradientTape block is used to compute gradients for the actor and critic networks.
Agent chooses an action based on the actor’s output probabilities and takes that action in the environment.
It observes the next state, reward, and whether the episode is done.
Advantage function is computed, which is the difference between the expected return and the estimated value at the current state.
Actor and Critic losses are calculated based on the advantage function.
Gradients are computed using tape.gradient and then applied to update the actor and critic networks using the respective optimisers.
Episode’s total reward is updated, and the loop continues until the episode ends.
Every 10 episodes, the current episode number and reward are printed.

Python

# Main training loop
num_episodes = 1000
gamma = 0.99

for episode in range(num_episodes):
    state = env.reset()
    episode_reward = 0

    with tf.GradientTape(persistent=True) as tape:
        for t in range(1, 10000):  # Limit the number of time steps
            # Choose an action using the actor
            action_probs = actor(np.array([state]))
            action = np.random.choice(env.action_space.n, p=action_probs.numpy()[0])

            # Take the chosen action and observe the next state and reward
            next_state, reward, done, _ = env.step(action)

            # Compute the advantage
            state_value = critic(np.array([state]))[0, 0]
            next_state_value = critic(np.array([next_state]))[0, 0]
            advantage = reward + gamma * next_state_value - state_value

            # Compute actor and critic losses
            actor_loss = -tf.math.log(action_probs[0, action]) * advantage
            critic_loss = tf.square(advantage)

            episode_reward += reward

            # Update actor and critic
            actor_gradients = tape.gradient(actor_loss, actor.trainable_variables)
            critic_gradients = tape.gradient(critic_loss, critic.trainable_variables)
            actor_optimizer.apply_gradients(zip(actor_gradients, actor.trainable_variables))
            critic_optimizer.apply_gradients(zip(critic_gradients, critic.trainable_variables))

            if done:
                break

    if episode % 10 == 0:
        print(f&quot;Episode {episode}, Reward: {episode_reward}&quot;)

env.close()

Output:

Episode 0, Reward: 29.0
Episode 10, Reward: 14.0
Episode 20, Reward: 15.0
Episode 30, Reward: 15.0
Episode 40, Reward: 31.0
Episode 50, Reward: 20.0
Episode 60, Reward: 22.0
Episode 70, Reward: 8.0
Episode 80, Reward: 51.0
Episode 90, Reward: 14.0
Episode 100, Reward: 11.0
Episode 110, Reward: 25.0
Episode 120, Reward: 16.0
....

Advantages of Actor Critic Algorithm

The Actor-Critic method offer several advantages:

Improved Sample Efficiency: The hybrid nature of Actor-Critic algorithms often leads to improved sample efficiency, requiring fewer interactions with the environment to achieve optimal performance.
Faster Convergence: The method’s ability to update both the policy and value function concurrently contributes to faster convergence during training, enabling quicker adaptation to the learning task.
Versatility Across Action Spaces: Actor-Critic architectures can seamlessly handle both discrete and continuous action spaces, offering flexibility in addressing a wide range of RL problems.
Off-Policy Learning (in some variants): Learns from past experiences, even when not directly following the current policy.

Advantage Actor Critic (A2C) vs. Asynchronous Advantage Actor Critic (A3C)

Asynchronous Advantage Actor-Critic (A3C) builds upon A2C by introducing parallelism.

In A2C, a single actor-critic pair interacts with the environment and updates its policy based on the experiences it gathers. However, A3C utilizes multiple actor-critic pairs operating simultaneously. Each pair interacts with a separate copy of the environment, collecting data independently. These experiences are then used to update a global actor-critic network.

Imagine training multiple agents simultaneously, each exploring a separate world. That’s the core idea behind A3C (Asynchronous Advantage Actor-Critic). These agents, called “workers,” independently learn from their experiences and update a central value function. This parallel approach allows A3C to explore the environment much faster than a single agent, leading to quicker learning.

A2C (Advantage Actor-Critic) is like A3C’s simpler cousin. It uses the same core concept of actor-critic with an advantage function, but without the parallel workers. While A2C explores the environment less extensively, studies have shown it can achieve similar performance to A3C while being easier to implement and requiring less computational power.

Applications of Actor Critic Algorithm

The Actor-Critic algorithm’s versatility extends its reach across a myriad of applications within the field of artificial intelligence. Some notable applications include:

Robotics: Actor-Critic algorithms empower robots to learn optimal control policies, allowing them to adapt and navigate complex environments.
Game Playing: In the realm of gaming, the Actor-Critic method proves valuable for training agents to make strategic decisions, enhancing their gameplay over time.
Autonomous Vehicles: The hybrid nature of Actor-Critic algorithms makes them suitable for training autonomous vehicles to make dynamic decisions in real-time, contributing to the evolution of self-driving technology.
Finance and Trading: Reinforcement learning, particularly Actor-Critic approaches, is employed to optimize trading strategies and make intelligent financial decisions in dynamic markets.
Healthcare: Actor-Critic methods can be applied to personalized treatment planning, where agents learn to make decisions that maximize patient outcomes based on individual health profiles.

Conclusion

In conclusion, the Actor-Critic algorithm emerges as a pivotal advancement in reinforcement learning, effectively addressing challenges faced by traditional RL algorithms.