Prerequisite: ML | Linear Regression
Linear regression is a supervised learning algorithm used for computing linear relationships between input (X) and output (Y).
The steps involved in ordinary linear regression are:
Training phase: Compute to minimize the cost.
Predict output: for given query point ,
As evident from the image below, this algorithm cannot be used for making predictions when there exists a non-linear relationship between X and Y. In such cases, locally weighted linear regression is used.
Locally Weighted Linear Regression:
Locally weighted linear regression is a non-parametric algorithm, that is, the model does not learn a fixed set of parameters as is done in ordinary linear regression. Rather parameters are computed individually for each query point . While computing , a higher “preference” is given to the points in the training set lying in the vicinity of than the points lying far away from .
The modified cost function is:
where, is a non-negative “weight” associated with training point .
For s lying closer to the query point , the value of is large, while for s lying far away from the value of is small.
A typical choice of is:
where, is called the bandwidth parameter and controls the rate at which falls with distance from
Clearly, if is small is close to 1 and if is large is close to 0.
Thus, the training-set-points lying closer to the query point contribute more to the cost than the points lying far away from .
For example –
Consider a query point = 5.0 and let and be two points in the training set such that = 4.9 and = 3.0.
Using the formula with = 0.5:
Thus, the weights fall exponentially as the distance between and increases and so does the contribution of error in prediction for to the cost.
Consequently, while computing , we focus more on reducing for the points lying closer to the query point (having larger value of ).
Steps involved in locally weighted linear regression are:
Compute to minimize the cost.
Predict Output: for given query point ,
Points to remember:
- Locally weighted linear regression is a supervised learning algorithm.
- It a non-parametric algorithm.
- There exists No training phase. All the work is done during the testing phase/while making predictions.
Attention reader! Don’t stop learning now. Get hold of all the important CS Theory concepts for SDE interviews with the CS Theory Course at a student-friendly price and become industry ready.
- Implementation of Locally Weighted Linear Regression
- ML | Linear Regression vs Logistic Regression
- Linear Regression (Python Implementation)
- Multiple Linear Regression using R
- Linear Regression using PyTorch
- Simple Linear-Regression using R
- Linear Regression Using Tensorflow
- ML | Linear Regression
- Gradient Descent in Linear Regression
- Mathematical explanation for Linear Regression working
- ML | Boston Housing Kaggle Challenge with Linear Regression
- ML | Normal Equation in Linear Regression
- ML | Multiple Linear Regression using Python
- ML | Rainfall prediction using Linear regression
- A Practical approach to Simple Linear Regression using R
- Python | Linear Regression using sklearn
- ML | Multiple Linear Regression (Backward Elimination Technique)
- Pyspark | Linear regression with Advanced Feature Dataset using Apache MLlib
- Polynomial Regression for Non-Linear Data - ML
- ML - Advantages and Disadvantages of Linear Regression
If you like GeeksforGeeks and would like to contribute, you can also write an article using contribute.geeksforgeeks.org or mail your article to firstname.lastname@example.org. See your article appearing on the GeeksforGeeks main page and help other Geeks.
Please Improve this article if you find anything incorrect by clicking on the "Improve Article" button below.