Prerequisite: ML | Linear Regression

Linear regression is a supervised learning algorithm used for computing linear relationships between input (X) and output (Y).

**The steps involved in ordinary linear regression are:**

Training phase:Compute to minimize the cost.

Predict output:for given query point ,

As evident from the image below, this algorithm cannot be used for making predictions when there exists a non-linear relationship between X and Y. In such cases, locally weighted linear regression is used.

### Locally Weighted Linear Regression:

Locally weighted linear regression is a non-parametric algorithm, that is, the model does not learn a fixed set of parameters as is done in ordinary linear regression. Rather parameters are computed individually for each query point . While computing , a higher “preference” is given to the points in the training set lying in the vicinity of than the points lying far away from .

The modified cost function is:

where, is a non-negative “weight” associated with training point .

For s lying closer to the query point , the value of is large, while for s lying far away from the value of is small.

A typical choice of is:

where, is called the bandwidth parameter and controls the rate at which falls with distance from

Clearly, if is small is close to 1 and if is large is close to 0.

Thus, the training-set-points lying closer to the query point contribute more to the cost than the points lying far away from .

**For example – **

Consider a query point = 5.0 and let and be two points in the training set such that = 4.9 and = 3.0.

Using the formula with = 0.5:

Thus, the weights fall exponentially as the distance between and increases and so does the contribution of error in prediction for to the cost.

Consequently, while computing , we focus more on reducing for the points lying closer to the query point (having larger value of ).

**Steps involved in locally weighted linear regression are:**

Compute to minimize the cost.

Predict Output:for given query point ,

**Points to remember:**

- Locally weighted linear regression is a supervised learning algorithm.
- It a non-parametric algorithm.
- There exists No training phase. All the work is done during the testing phase/while making predictions.