Gradient Descent :
Gradient descent is an optimization algorithm used to find the values of parameters of a function that minimizes a cost function. It is an iterative algorithm. We use gradient descent to update the parameters of the model. Parameters refer to coefficients in Linear Regression and weights in neural networks.
Gradient descent can also converge even if the learning rate is kept fixed.
Normal Equation :
Normal Equation, an analytical approach used for optimization. It is an alternative for Gradient descent. Normal equation performs minimization without iteration.
Normal equations are equations obtained by setting equal to zero the partial derivatives of the sum of squared errors or cost function; normal equations allow one to estimate the parameters of multiple linear regression.
X = input features value
y = output value
If the term X is non-invertible or singular then we can use regularization.
Difference between Gradient Descent and Normal Equation.
|S.NO.||Gradient Descent||Normal Equation|
|1.||In gradient descenet , we need to choose learning rate.||In normal equation , no need to choose learning rate.|
|2.||It is an iterative algorithm.||It is analytical approach.|
|3.||Gradient descent works well with large number of features.||Normal equation works well with small number of features.|
|4.||Feature scaling can be used.||No need for feature scaling.|
|5.||No need to handle non-invertibility case.||If (X) is non-invertible , regularization can be used to handle this.|
|6.||Algorithm complexity is O(k).
n is the number of features.
|Algorithm complexity is O().
n is the number of features.
- Difference between Batch Gradient Descent and Stochastic Gradient Descent
- Difference between Recursive Predictive Descent Parser and Non-Recursive Predictive Descent Parser
- Gradient Descent algorithm and its variants
- ML | Stochastic Gradient Descent (SGD)
- Optimization techniques for Gradient Descent
- Gradient Descent in Linear Regression
- ML | Mini-Batch Gradient Descent with Python
- ML | Normal Equation in Linear Regression
- Difference between Normal Mouse and Gaming Mouse
- Difference between Normal Processor and AI Processor
- Multivariate Optimization - Gradient and Hessian
- ML | Momentum-based Gradient Optimizer introduction
- ML | XGBoost (eXtreme Gradient Boosting)
- LightGBM (Light Gradient Boosting Machine)
- ML - Gradient Boosting
- SciPy - Integration of a Differential Equation for Curve Fit
- Handwritten Equation Solver in Python
- Difference and Similarities between PHP and C
- Difference between Stop and Wait protocol and Sliding Window protocol
- Similarities and Difference between Java and C++
If you like GeeksforGeeks and would like to contribute, you can also write an article using contribute.geeksforgeeks.org or mail your article to firstname.lastname@example.org. See your article appearing on the GeeksforGeeks main page and help other Geeks.
Please Improve this article if you find anything incorrect by clicking on the "Improve Article" button below.