In machine learning, gradient descent is an optimization technique used for computing the model parameters (coefficients and bias) for algorithms like linear regression, logistic regression, neural networks, etc. In this technique, we repeatedly iterate through the training set and update the model parameters in accordance with the gradient of error with respect to the training set.
Depending on the number of training examples considered in updating the model parameters, we have 3-types of gradient descents:
- Batch Gradient Descent: Parameters are updated after computing the gradient of error with respect to the entire training set
- Stochastic Gradient Descent: Parameters are updated after computing the gradient of error with respect to a single training example
- Mini-Batch Gradient Descent: Parameters are updated after computing the gradient of error with respect to a subset of the training set
|Batch Gradient Descent||Stochastic Gradient Descent||Mini-Batch Gradient Descent|
|Since entire training data is considered before taking a step in the direction of gradient, therefore it takes a lot of time for making a single update.||Since only a single training example is considered before taking a step in the direction of gradient, we are forced to loop over the training set and thus cannot exploit the speed associated with vectorizing the code.||Since a subset of training examples is considered, it can make quick updates in the model parameters and can also exploit the speed associated with vectorizing the code.|
|It makes smooth updates in the model parameters||It makes very noisy updates in the parameters||Depending upon the batch size, the updates can be made less noisy – greater the batch size less noisy is the update|
Thus, mini-batch gradient descent makes a compromise between the speedy convergence and the noise associated with gradient update which makes it a more flexible and robust algorithm.
Mini-Batch Gradient Descent:
Let theta = model parameters and max_iters = number of epochs.
for itr = 1, 2, 3, …, max_iters:
for mini_batch (X_mini, y_mini):
- Forward Pass on the batch X_mini:
- Make predictions on the mini-batch
- Compute error in predictions (J(theta)) with the current values of the parameters
- Backward Pass:
- Compute gradient(theta) = partial derivative of J(theta) w.r.t. theta
- Update parameters:
- theta = theta – learning_rate*gradient(theta)
Below is the Python Implementation:
Step #1: First step is to import dependencies, generate data for linear regression and visualize the generated data. We have generated 8000 data examples, each having 2 attributes/features. These data examples are further divided into training set (X_train, y_train) and testing set (X_test, y_test) having 7200 and 800 examples respectively.
Number of examples in training set = 7200
Number of examples in testing set = 800
Step #2: Next, we write the code for implementing linear regression using mini-batch gradient descent.
gradientDescent() is the main driver function and other functions are helper functions used for making predictions –
hypothesis(), computing gradients –
gradient(), computing error –
cost() and creating mini-batches –
create_mini_batches(). The driver function initializes the parameters, computes the best set of parameters for the model and returns these parameters along with a list containing history of errors as the parameters get updated.
gradientDescent() function to compute the model parameters (theta) and visualizing the change in error function.
Bias = [0.81830471]
Coefficients = [[1.04586595]]
Step #3 : Finally, we make predictions on the testing set and compute the mean absolute error in predictions.
Mean absolute error = 0.4366644295854125
The orange line represents the final hypothesis function: theta + theta*X_test[:, 1] + theta*X_test[:, 2] = 0
Attention reader! Don’t stop learning now. Get hold of all the important CS Theory concepts for SDE interviews with the CS Theory Course at a student-friendly price and become industry ready.
- Difference between Batch Gradient Descent and Stochastic Gradient Descent
- ML | Stochastic Gradient Descent (SGD)
- Optimization techniques for Gradient Descent
- Gradient Descent in Linear Regression
- Gradient Descent algorithm and its variants
- Difference between Gradient descent and Normal equation
- Vectorization Of Gradient Descent
- PyQt5 QSpinBox - Getting descent of the font
- Python | Plotting an Excel chart with Gradient fills using XlsxWriter module
- Python | Morphological Operations in Image Processing (Gradient) | Set-3
- How to find Gradient of a Function using Python?
- Python - tensorflow.GradientTape.gradient()
- ML | Momentum-based Gradient Optimizer introduction
- ML | XGBoost (eXtreme Gradient Boosting)
- PyQt5 - Gradient color Bar of Progress Bar
- PyQt5 - Gradient Color Progress Bar
- LightGBM (Light Gradient Boosting Machine)
- Multivariate Optimization - Gradient and Hessian
- ML - Gradient Boosting
- Make a gradient color mapping on a specified column in Pandas
If you like GeeksforGeeks and would like to contribute, you can also write an article using contribute.geeksforgeeks.org or mail your article to email@example.com. See your article appearing on the GeeksforGeeks main page and help other Geeks.
Please Improve this article if you find anything incorrect by clicking on the "Improve Article" button below.