Classification of Neural Network in TensorFlow

Classification is used for feature categorization, and only allows one output response for every input pattern as opposed to permitting various faults to occur with a specific set of operating parameters. The category that has the greatest output value is chosen by the classification network. When integrated with numerous forms of predictive neural networks in a hybrid system, classification neural networks become incredibly powerful.

What is Artificial Neural Network?

A substantial majority of heavily integrated processing components known as neurons comprise artificial neural networks, which are computational models that were modeled after biological neural networks.

• Pattern recognition or data segmentation are two instances of applications for which an ANN (Artificial Neural network) is configured.
• It is equipped to make sense of enormous or perplexing material.
• It extracts patterns and detects trends that are too subtle for individuals or other computer technologies to grab on.

Activation Function

The weights and the input-output function which are specified for the unit impact how well the ANN (Artificial Neural Network) responds. One of the three categories outlined below best describes this function:

• Linear:  The linear units The output activity has a linear relationship with the overall grade output.        
• Threshold:  Depending on whether the total input is stronger than or lesser than a certain threshold value, the output is set at one of two levels.
• Sigmoid:  As the input fluctuates, the output varies continuously but not linearly. Whereas sigmoid units are more roughly related to actual neurons than the threshold or linear units, all three must be considered approximations.
• ReLU:  The rectified linear activation function, commonly known as ReLU, is a non-linear or piecewise linear function that, if the input is positive, gives the input directly; if not, it outputs zero.
• Step:  The rectified linear activation function, popularly known as ReLU, is a non-linear or piecewise linear function that, if the input is positive, gives the input directly; if not, it gives zero.
• SoftMax:  It turns a real number into a probability density that adds to one, with each member in the output vector reflecting the likelihood that the input vector belongs to a given category. 
• Hyperbolic/tanh:  A potential function that could be utilized as a nonlinear activation function within layers of a neuron is the hyperbolic function. In fact, it has some similarities to the sigmoid activation function. Both resemble each other a lot. Tanh, however, will map inputs to a range from -1 and 1, whereas a sigmoid function will map input parameters to a range of 0 and 1.                         

Implementation of sigmoid function :

• Perform an operation that uses the sigmoid function.
• To demonstrate this, we used scikit-learn to create a classification example.
• The ‘make_blobs’ function is used to create a dataset with 60 samples and 2 features.
• Perform operations for classification like binary, to display a number of blobs.

Python3

# import library for code implementation
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline

# activation function -> sigmoid


def sigmoid(x):
    return 1/(1+np.exp(-x))


x_sample = np.linspace(-20, 20, 50)
z_sample = sigmoid(x_sample)

# To display graph
plt.plot(x_sample, z_sample)

Output:

Let’s look at a demonstration of categorization:

• Scikit-learn has significant features and the capability to generate data sets for us.
•  Then we’ll declare that our data is equivalent to blobs. 
• There are only a few blobs that are generated there that we can characterize. 
• As a result, this is merely a binary classification issue as we need to generate 100 samples and three blobs from the number of features.

1. Undertake sigmoid function operations, import libraries, and describe blob characteristics for dataframe in order to display the dataframe.

Python3

# Import library to create blobs
from sklearn.datasets import make_blobs

# Perform operations to elevate sigmoid function
class Implementation():
    def __init__(self, x):
        super().__init__([x])

        def solve(self, x_val):
            return 1/(1+np.exp(-x))


# Define features of blobs ie.
# its size, center, random_state and features
Dataframe = make_blobs(n_samples=60, n_features=2,
                       centers=2, random_state=85)

# print Dataframe
Dataframe

Output:

2. To obtain the output of the type of dataframe we are using in this example, employ the type function.

Python3

# To display type of dataframe
type(Dataframe)

Output:

tuple

3. Indexing as one to retrieve the outcome of the dataframe’s first index with its values

Python3

# Print Dataframe at first index
Dataframe[1]

Output:

array([0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 0, 1, 0, 1,
       0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 1,
       0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1])

4. Generate the scatterplot and configure the features as well as label and then produce the plots of blobs.

Python3

# Create the scatterplot of features
features = Dataframe[0]
labels = Dataframe[1]

# Display the scatter plot of blobs
plt.scatter(features[:, 0], features[:, 1], c = labels, cmap = "cool")

Output:

Implementation of SoftMax activation function

• In neural networks, the softmax activation function is very often adopted for multi-class classification methods. 
• It is a subtype of activation function that transforms a real-number vector into a probability distribution with a total of one. 
• In a multi-class classification issue, the outcome of the softmax function could well be regarded as the likelihood of each class.
• The softmax function is distinguishable, which permits the use of gradient-based optimization techniques, such as stochastic gradient descent.

1. The first and foremost step would be importing the library, then constructing a function to retrieve the softmax function’s output.

Python3

# Importing library
import numpy as np

# This will return the output of softmax function
def softmax(x):
  res_x = np.exp(x)
  sum = res_x.sum()

  #using round function in result
  softmax_function_x = np.round(res_x/sum,3)
  return softmax_function_x

# initializing x with certain values
x = [0.54, 2.23, -0.6]
softmax(x)

Output:

array([0.148, 0.804, 0.047])

2. Devise a separate method to perform sum by division and thereafter return the result

Python3

# Function named division_sum to return output
def division_sum(x):
  sum_x = np.sum(x)

  #using round function to get output
  output_x = np.round(x/sum_x,3)
  return output_x

# Initializing x1
x1 = [0.54, 2.23, -0.6]
division_sum(x1)

Output:

array([ 0.249,  1.028, -0.276])

3. Further To acquire the result in the form of an array, construct a variable with the names x1 and x2 and execute the function division sum one which was previously defined.

Python3

# Initializing x2
x2 = [-0.36,0.2,-0.52]
division_sum(x2)

Output:

array([ 0.529, -0.294,  0.765])

Python3

# import required library
import numpy as np
import matplotlib.pyplot as plt
import numpy as np

plt.style.use('seaborn')
plt.figure(figsize=(9,5))
def softmax(x):
    return np.exp(x) / np.sum(np.exp(x))
x = np.linspace(-6, 6)

# To plot the graph of softmax activation function
plt.plot(x, softmax(x))
plt.title('Softmax activation')
plt.show()

Output:

Implementation of rectified linear activation function 

• The rectified linear activation function is simple to implement in Python. 
• The much more straightforward method is to use the max() function.
• We predict that the procedure will discard whatever favorable input data.
• Intact, however any input parameters of 0.0 or below should be changed to 0.0.

Python3

# Rectified linear function
def rectified_activation(z):
    return max(0.0, z)

# Whenever we execute the code, we see that positive numbers are provided in their entirety,however negative parameters
# are trimmed to 0.0.

# Illustration with a positive input
z = 2.0
print('rectified_activation(%.2f) gives output as %.2f' %
      (z, rectified_activation(z)))
z = 2000.0
print('rectified_activation(%.2f) gives output as %.2f' %
      (z, rectified_activation(z)))

# Illustration with a zero input
z = 0.0
print('rectified_activation(%.2f) gives output as %.2f' %
      (z, rectified_activation(z)))

#  Illustration with a negative input
z = -2.0
print('rectified_activation(%.2f) gives output as %.2f' %
      (z, rectified_activation(z)))
z = -2000.0
print('rectified_activation(%.2f) gives output as %.2f' %
      (z, rectified_activation(z)))

Output:

rectified_activation(2.00) gives output as 2.00
rectified_activation(2000.00) gives output as 2000.00
rectified_activation(0.00) gives output as 0.00
rectified_activation(-2.00) gives output as 0.00
rectified_activation(-2000.00) gives output as 0.00

By charting a succession of inputs and computed outputs, we may acquire a notion of the function’s connection among both outputs and inputs.

Python3

# plot the outcome of given input and output
from matplotlib import pyplot

# rectified linear activation function
def rectified(z):
 return max(0.0, z)

# Demonstrating a sequence of inputs
sequence_in = [z for z in range(-11, 12)]

# calculate the outcome
sequence_out = [rectified_activation(z) for z in sequence_in]

# line plot
pyplot.plot(sequence_in, sequence_out)
pyplot.show()

#The illustration creates a sequence of numbers ranging from -11 to 12 and computes the
#rectified linear activation for every source before displaying the outcome.

Output:

Implementation of Step activation function 

• Neural network models, which are basic neural networks that can categorize inputs based on a set of training instances, may utilize it to categorize responses. It can be challenging to employ the step function in some kinds of neural networks that depend on gradient-based methodologies since it is not distinguishable at the cutoff.
• Instead, the continuous and distinguishable sigmoid function, a version of the step function, is frequently utilized.
• A piecewise function can be used to describe the step function, with a separate function defining the output for each range of input values.
• The step function may be used in conjunction with additional activation functions to build more sophisticated neural networks that are capable of handling a larger variety of tasks.

Python3

# Binary step activation function

#It returns '0' if the input is the less than zero
#It returns '1' if the input is greater than zero

# Import the library
import numpy as np
import matplotlib.pyplot as plt
import numpy as np

# It returns '0' is the input is less than zero otherwise it returns one
def Step_activation_function(y):
    return np.heaviside(y,1)

#Plot the step activation function
y = np.linspace(-8, 8)
plt.plot(y, Step_activation_function(y))
plt.axis('tight')

#Give the graph title as per your choice
plt.title('Step Activation')
plt.show()

Output:

graph of step activation function

Implementation of Hyperbolic/tanh activation function

• Several neural network topologies choose the tanh function because it is a continuous and distinguishable function.
• The tanh function has a symmetric shape around the origin, therefore for negative input parameters, it represents a negative outcome, and for positive input values, it is a positive output.
• The tanh function is much more responsive to changes in the input because it is steeper than that of the sigmoid function in the center of its spectrum.
• The tanh function is frequently employed in recurrent neural networks and may be utilized in neural networks’ convolution neurons as well as their output neurons.
• Since it can aid in preventing the vanishing gradient problem, which can happen in deep neural networks, the hyperbolic activation function is a common option for neural networks.

Python3

#Since its values typically range from -1 to 1, the average for the hidden
#neuron of a neural network will be 0 or quite near to that though.
#This serves to centre the input by getting the mean near to 0.
#This greatly simplifies understanding for the successive layer.


# Importing library
import matplotlib.pyplot as plt
import numpy as np

def Hyperbolic_or_tanh(z):

    # define the formula used for hyperbolic functions
    t = (np.exp(z)-np.exp(-z))/(np.exp(z)+np.exp(-z))
    dt = 1-t**2
    return t, dt

y = np.arange(-2,2,0.01)
Hyperbolic_or_tanh(y)[0].size, Hyperbolic_or_tanh(y)[1].size

Output:

(400, 400)

Python3

#As a result of its ranging to be between -1 and 1, or -1 output 1,
#the outcome has now become zero-centered.


# Setup centered axes
figure, axe = plt.subplots(figsize=(8, 6))
axe.spines['left'].set_position('center')
axe.spines['bottom'].set_position('center')
axe.spines['right'].set_color('none')
axe.spines['top'].set_color('none')
axe.xaxis.set_ticks_position('bottom')
axe.yaxis.set_ticks_position('left')


#As improvement is simpler with this approach, it is frequently favoured
#in practise as compared to others.

# Create and show plot
axe.plot(y, Hyperbolic_or_tanh(y)[0], color="#104AC7", linewidth=2, label="tanh_or_hyperbolic")
axe.plot(y, Hyperbolic_or_tanh(y)[1], color="#6621E2", linewidth=2, label="output_derivative")
axe.legend(loc="upper right", frameon=False)

# Display the hyperbolic graph
figure.show()

Output:

graph of hyperbolic activation function

Conclusion: 

To summarize, classification neural networks are used to categorize characteristics and only permit one output response for each input pattern. When used in a hybrid system with other prognostic neural networks. The weights and the input-output function of each unit, which might be linear, threshold, sigmoid, RelU, step, Hyperbolic/tanh, or SoftMax, influence the performance of an ANN. For neural network categorization, you may utilize a variety of activation functions.