# Hopfield Neural Network

**Prerequisites:** RNN

The Hopfield Neural Networks, invented by Dr John J. Hopfield consists of one layer of *‘n’* fully connected recurrent neurons. It is generally used in performing auto association and optimization tasks. It is calculated using a converging interactive process and it generates a different response than our normal neural nets.

**Discrete Hopfield Network: **It is a fully interconnected neural network where each unit is connected to every other unit. It behaves in a discrete manner, i.e. it gives finite distinct output, generally of two types:

**Binary (0/1)****Bipolar (-1/1)**

The weights associated with this network is symmetric in nature and has the following properties.

**Structure & Architecture **

- Each neuron has an inverting and a non-inverting output.
- Being fully connected, the output of each neuron is an input to all other neurons but not self.

**Fig 1** shows a sample representation of a Discrete Hopfield Neural Network architecture having the following elements.

Input to the n given neurons._{[ x1 , x2 , ... , xn ] ->}Output obtained from the n given neurons_{[ y1 , y2 , ... , yn ] ->}Wweight associated with the connection between the i_{ij ->}^{th}and the j^{th}neuron.

**Training Algorithm**

For storing a set of input patterns S(p) [p = 1 to P], where **S(p) = S _{1}(p) … S_{i}(p) … S_{n}(p)**, the weight matrix is given by:

**For binary patterns**

**For bipolar patterns**

(i.e. weights here have no self connection)

**Steps Involved**

Step 1 -Initialize weights(wto store patterns_{ij)}(using training algorithm).

Step 2 -For each input vector y_{i}, performsteps 3-7.

Step 3 -Make initial activators of the network equal to the external input vector x.

Step 4 -For each vector y_{i}, performsteps 5-7.

Step 5 -Calculate the total input of the networkyusing the equation given below._{in}

Step 6 -Apply activation over the total input to calculate the output as per the equation given below:

(where **θ _{i}** (threshold) and is normally taken as 0)

Step 7 -Now feedback the obtained output y_{i }to all other units. Thus, the activation vectors are updated.

Step 8 -Test the network for convergence.

**Example Problem**

Consider the following problem. We are required to create Discrete Hopfield Network with bipolar representation of input vector as [1 1 1 -1] or [1 1 1 0] (in case of binary representation) is stored in the network. Test the hopfield network with missing entries in the first and second component of the stored vector (i.e. [0 0 1 0]).

**Step by Step Solution**

Step 1 -given input vector, x = [1 1 1 -1] (bipolar) and we initialize the weight matrix(was:_{ij)}

and weight matrix with no self connection is:

Step 3 -As per the question, input vector x with missing entries, x = [0 0 1 0] ([x1 x2 x3 x4]) (binary)-Make y_{i}= x = [0 0 1 0]([y1 y2 y3 y4])

Step 4 -Choosing unit y_{i}(order doesn't matter)for updating its activation.-Take the i^{th}column of the weight matrix for calculation.(we will do the next steps for all values of yi and check if there is convergence or not)

‘

now for next unit, we will take updated value via feedback. (i.e. y = [1 0 1 0])

now for next unit, we will take updated value via feedback.(i.e. y = [1 0 1 0])

now for next unit, we will take updated value via feedback.(i.e. y = [1 0 1 0])

**Continuous Hopfield Network:** Unlike the discrete hopfield networks, here the time parameter is treated as a continuous variable. So, instead of getting binary/bipolar outputs, we can obtain values that lie between 0 and 1. It can be used to solve constrained optimization and associative memory problems. The output is defined as:

where,voutput from the continuous hopfield network_{i =}uinternal activity of a node in continuous hopfield network._{i =}

**Energy Function**

The hopfield networks have an energy function associated with them. It either diminishes or remains unchanged on update (feedback) after every iteration. The energy function for a continuous hopfield network is defined as:

To determine if the network will converge to a stable configuration, we see if the energy function reaches its minimum by:

The network is bound to converge if the activity of each neuron wrt time is given by the following differential equation: