What Are the Cases Where It Is Fine to Initialize All Weights to Zero?

Answer: It’s suitable for biases, not for weights, as it hinders learning by making neuron outputs identical.

Initializing all weights to zero in neural networks is a topic of nuanced consideration, primarily due to how it affects the learning process.

Aspect	Zero Initialization	Non-Zero Initialization
Learning Dynamics	Leads to symmetrical neuron behavior, causing neurons to learn the same features.	Ensures diverse neuron behavior, allowing for richer feature learning.
Gradient Descent	With identical weights, gradients for all neurons in a layer are the same, hindering learning.	Different initial weights lead to different gradients, facilitating learning.
Application	Can be used for bias initialization without impairing learning.	Preferred for weights to break symmetry and enable effective network training.
Suitable Scenarios	Single-layer networks or linear models where symmetry is not an issue.	Multi-layer networks, where breaking symmetry is crucial for learning.

Conclusion

In practice, initializing all weights to zero in neural networks is largely inadvisable due to the critical issue of symmetry breaking. When weights are initialized to zero, it causes neurons within the same layer to follow identical update paths during training, thus learning the same features and rendering the network’s capacity equivalent to a single neuron model. This severely limits the network’s ability to model complex patterns and reduces its overall effectiveness.

However, zero initialization for biases is generally acceptable as it does not contribute to the symmetry problem in the same way weights do. Biases can start from zero since their primary role is to provide an adjustable threshold for neuron activation rather than to diversify learning paths.

For weights, it’s crucial to employ strategies that introduce asymmetry from the onset, such as small random values, which allow for efficient and diverse learning across neurons. This approach ensures that each neuron can learn unique features, significantly enhancing the network’s overall capability to model complex relationships and patterns within the data.