Explanation of Fundamental Functions involved in A3C algorithm

Although any implementation of the Asynchronous Advantage Actor Critic algorithm is bound to be complex, all the implementations will have the one thing in common – the presence of the Global Network and the worker class.

1. Global Network class: This contains all the required Tensorflow operations to autonomously create the neural networks.
2. Worker class: This class is used to simulate the learning process of a worker who has its own copy of the environment and a “personal” neural network.

For the following implementation, the following modules will be required:-

1. Numpy
2. Tensorflow
3. Multiprocessing

The following lines of code denote the fundamental functions required to build the respective class.

Global Network class:

The following lines contain the various functions describe the member functions of the class defined above.

Initializing the class:

-> tf.placeholder()       - Inserts a placeholder for a tensor that will always be fed.
-> tf.reshape()           - Reshapes the input tensor
-> slim.conv2d()          - Adds an n-dimensional convolutional network
-> slim.fully_connected() - Adds a fully connected layer

Note the following definitions:

• Filter: It is a small matrix which is used to apply different effects on a given image.
• Padding: It is the process of adding an extra row or column on the boundaries of an image to completely compute the filter convolution value of the filter.
• Stride: It is the number of steps after which the filter is set upon the pixel in a given direction.

Building the Recurrent network:

-> tf.nn.rnn_cell.BasicLSTMCell()  - Builds a basic LSTM Recurrent network cell
-> tf.expand_dims()                - Inserts a dimension of 1 at the dimension index
                                     axis of input's shape
-> tf.shape()                      - returns the shape of the tensor
-> tf.nn.rnn_cell.LSTMStateTuple() - Creates a tuple to be used by
                                     the LSTM cells for state_size,
                                     zero_state and output state.
-> tf.nn.dynamic_rnn()             - Builds a Recurrent network according to
                                     the Recurrent network cell

Building the output layers for value and policy estimation:

Building the Master network and deploying the workers:

Running the parallel Tensorflow operations:

-> tf.Session()                    - A class to run the Tensorflow operations
-> tf.train.Coordinator()          - Returns a coordinator for the multiple threads
-> tf.train.get_checkpoint_state() - Returns a valid checkpoint state
                                     from the "checkpoint" file
-> saver.restore()                 - Is used to store and restore the models
-> sess.run()                      - Outputs the tensors and metadata obtained
                                     from running a session

Updating the global network parameters:

-> tf.one_hot()              - Returns a one-hot encoded tensor
-> tf.reduce_sum()           - Reduces the input tensor along the input dimensions
-> tf.gradients()            - Constructs the symbolic derivatives of the sum
-> tf.clip_by_global_norm()  - Performs the clipping of values of the multiple tensors
                               by the ratio of the sum of the norms
-> trainer.apply_gradients() - Performs the update step according to the optimizer.

Defining a utility function to copy the parameters of one network to the other:

-> tf.get_collection() - Returns the list of values in the collection with the given name.

Worker Class:

Defining the Class:

The following lines of code describes the member functions of the above described class

Initializing the class:

Defining the function for the worker to interact with it’s environment:

-> sess.as_default() - Sets the current session as the default session
-> self.env.new_episode() - Initializes a new training episode for the worker

Defining the training function for the worker:

AlindGupta

Check out this Author's contributed articles.