Open In App
Related Articles

Multivariate Regression

Improve Article
Save Article
Like Article

Prerequisite Article-Machine Learning

The goal in any data analysis is to extract from raw information the accurate estimation. One of the most important and common question concerning if there is statistical relationship between a response variable (Y) and explanatory variables (Xi). An option to answer this question is to employ regression analysis in order to model its relationship. Further it can be used to predict the response variable for any arbitrary set of explanatory variables.

 The Problem:

Multivariate Regression is one of the simplest Machine Learning Algorithm. It comes under the class of Supervised Learning Algorithms i.e, when we are provided with training dataset. Some of the problems that can be solved using this model are:

  • A researcher has collected data on three psychological variables, four academic variables (standardized test scores), and the type of educational program the student is in for 600 high school students. She is interested in how the set of psychological variables is related to the academic variables and the type of program the student is in.
  • A doctor has collected data on cholesterol, blood pressure, and weight.  She also collected data on the eating habits of the subjects (e.g., how many ounces of red meat, fish, dairy products, and chocolate consumed per week).  She wants to investigate the relationship between the three measures of health and eating habits.
  • A property dealer wants to set housing prices which are based various factors like Size of house, No of bedrooms, Age of house, etc. We shall discuss the algorithm further using this example.

The Solution:

          The solution is divided into various parts.

  • Selecting the features: Finding the features on which a response variable depends (or not) is one of the most important steps in Multivariate Regression. To make our analysis simple, we assume that the features on which the response variable is dependent are already selected.
  • Normalizing the features: The features are then scaled in order to bring them in range of (0,1) to make better analysis. This can be done by changing the value of each feature by:
    \begin{matrix} Xi=\frac {x_i-\mu _i}{\delta _i}, Where,x_i=Training\ examples\ for\ ith\ feature, \\ \mu _i=mean\ of\ ith\ feature. \\  \delta _i=range \ of\ ith \ feature. \end{matrix}

  • Selecting Hypothesis and Cost function: A hypothesis is a predicted value of the response variable represented by h(x). Cost function defines the cost for wrongly predicting hypothesis. It should be as small as possible. We choose hypothesis function as linear combination of features X.
  • \begin{matrix} h(x^i)=\theta _0 +\theta _1x_1^i+........+\theta _nx_n^i \\ where\Theta =[\theta _0 +\theta _1+........+\theta _n]^Tis\ the \ parameter\ vector, \\  and \ x_i^j=value\ of \ ith\ feature\ in \ jth\ training\ example. \\ And\ the\ cost\ function\ as \ sum\ of\ squared\ error\ over\ all\ training\ examples.\\ J(\theta )=\frac {1}{2m*\sum (h_\theta(x^i)-y^i)^2}\end{matrix}

  • Minimizing the Cost function: Next some Cost minimization algorithm runs over the datasets which adjusts the parameters of the hypothesis. Once the cost function is minimized for the training dataset, it should also be minimized for an arbitrary dataset if the relation is universal. Gradient descent algorithm is a good choice for minimizing the cost function in case of multivariate regression.
  • Testing the hypothesis: The hypothesis function is then tested over the test set to check its correctness and efficiency.


Multivariate regression technique can be implemented efficiently with the help of matrix operations. With python, it can be implemented using “numpy” library which contains definitions and operations for matrix object.

The code requires “numpy” library for python which is not installed on GfG servers and thus the code is unable to run on gfg IDE. However link to the code is:


If you like GeeksforGeeks and would like to contribute, you can also write an article and mail your article to See your article appearing on the GeeksforGeeks main page and help other Geeks.

Whether you're preparing for your first job interview or aiming to upskill in this ever-evolving tech landscape, GeeksforGeeks Courses are your key to success. We provide top-quality content at affordable prices, all geared towards accelerating your growth in a time-bound manner. Join the millions we've already empowered, and we're here to do the same for you. Don't miss out - check it out now!

Last Updated : 19 Sep, 2023
Like Article
Save Article
Similar Reads