ML | Locally weighted Linear Regression
Prerequisite: ML | Linear Regression
Linear regression is a supervised learning algorithm used for computing linear relationships between input (X) and output (Y).
The steps involved in ordinary linear regression are:
Training phase: Compute
to minimize the cost.
Predict output: for given query point
,
As evident from the image below, this algorithm cannot be used for making predictions when there exists a non-linear relationship between X and Y. In such cases, locally weighted linear regression is used.
Locally Weighted Linear Regression:
Locally weighted linear regression is a non-parametric algorithm, that is, the model does not learn a fixed set of parameters as is done in ordinary linear regression. Rather parameters are computed individually for each query point
. While computing
, a higher “preference” is given to the points in the training set lying in the vicinity of
than the points lying far away from
.
The modified cost function is:
where, is a non-negative “weight” associated with training point
.
For s lying closer to the query point
, the value of
is large, while for
s lying far away from
the value of
is small.
A typical choice of is:
where, is called the bandwidth parameter and controls the rate at which
falls with distance from
Clearly, if is small
is close to 1 and if
is large
is close to 0.
Thus, the training-set-points lying closer to the query point contribute more to the cost
than the points lying far away from
.
For example –
Consider a query point = 5.0 and let
and
be two points in the training set such that
= 4.9 and
= 3.0.
Using the formula with
= 0.5:
Thus, the weights fall exponentially as the distance between and
increases and so does the contribution of error in prediction for
to the cost.
Consequently, while computing , we focus more on reducing
for the points lying closer to the query point (having larger value of
).
Steps involved in locally weighted linear regression are:
Compute
to minimize the cost.
Predict Output: for given query point,
Points to remember:
- Locally weighted linear regression is a supervised learning algorithm.
- It a non-parametric algorithm.
- There exists No training phase. All the work is done during the testing phase/while making predictions.