Effect of Bias in Neural Network
Neural Network is conceptually based on actual neuron of brain. Neurons are the basic units of a large neural network. A single neuron passes single forward based on input provided.
In Neural network, some inputs are provided to an artificial neuron, and with each input a weight is associated. Weight increases the steepness of activation function. This means weight decide how fast the activation function will trigger whereas bias is used to delay the triggering of the activation function.
For a typical neuron, if the inputs are x1, x2, and x3, then the synaptic weights to be applied to them are denoted as w1, w2, and w3.
y = f(x) = Σxiwi
where i is 1 to the number of inputs.
The weight shows the effectiveness of a particular input. More the weight of input, more it will have impact on network.
On the other hand Bias is like the intercept added in a linear equation. It is an additional parameter in the Neural Network which is used to adjust the output along with the weighted sum of the inputs to the neuron. Therefore Bias is a constant which helps the model in a way that it can fit best for the given data.
The processing done by a neuron is thus denoted as :
output = sum (weights * inputs) + bias
Need of bias
In above figure
y = mx+c
m = weight
c = bias
Now, Suppose if c was absent, then the graph will be formed like this:
Due to absence of bias, model will train over point passing through origin only, which is not in accordance with real-world scenario. Also with the introduction of bias, the model will become more flexible.
Suppose an activation function act() which get triggered on some input greater than 0.
input1 = 1
weight1 = 2
input2 = 2
weight2 = 2
output = input1*weight1 + input2*weight2
output = 6
suppose act(output) = 1
Now a bias is introduced in output as
bias = -6
the output become 0.
act(0) = 0
so activation function will not trigger.
Change in weight
Here in graph, as it can be seen that when:
- weight WI
changed from 1.0 to 4.0
- weight W2
changed from -0.5 to 1.5
On increasing the weight the steepness is increasing.
Therefore it can be inferred that
More the weight earlier activation function will trigger.
Change in bias
Here in graph below, when
Bias changed from -1.0 to -5.0
The change in bias is increasing the value of triggering activation function.
Therefore it can be inferred that from above graph that,
bias helps in controlling the value at which activation function will trigger.