TrisZaska's Machine Learning Blog

Why we need the bias term

1. Introduction
2. History and Overview about Artificial Neural Network
3. Single neural network
4. Multi-layer neural network
5. Install and using Multi-layer Neural Network to classify MNIST data
6. Summary
7. References

The bias term

Look at the pictures of Perceptron and the equation \((1)\) where we have extra \(w_0x_0\). It's call the bias, so what is it and why we definitely need it?
Firstly, it's constant value with \(x_0\) always equal 1 and \(w_0\) similar to another weight in Perceptron and we can also update \(w_0\), too. So, what is the role of the Bias in Neural Network?
The bias allow we to shift the Unit step function to the left or right to achieve successful learning. Usually in Perceptron, we set \(x_0 = 1\) and \(w_0 = -\theta\), so look at the images below to have an intuition about it
So, why we want to shift the Unit step function? Let's consider when we have all of the input equal to \(0\), but we want our Perceptron's output is \(1\). Adjust the weight is hopeless because \(\mathbf{z = w^Tx}\) always equal to \(0\). Then, the bias will come to solve this situation. That's why we definitely needed and the role of the bias in Multi-layer Neural Network is trying to shift the Sigmoid function instead of the Unit step function.

No comments :

Post a Comment

Leave a Comment...