 GeeksforGeeks App
Open App Browser
Continue

# Exploring Correlation in Python

This article aims to give a better understanding of a very important technique of multivariate exploration. A correlation Matrix is basically a covariance matrix. Also known as the auto-covariance matrix, dispersion matrix, variance matrix, or variance-covariance matrix. It is a matrix in which the i-j position defines the correlation between the ith and jth parameter of the given data set. When the data points follow a roughly straight-line trend, the variables are said to have an approximately linear relationship. In some cases, the data points fall close to a straight line, but more often there is quite a bit of variability of the points around the straight-line trend. A summary measure called correlation describes the strength of the linear association.

## Correlation in Python

Correlation summarizes the strength and direction of the linear (straight-line) association between two quantitative variables. Denoted by r, it takes values between -1 and +1. A positive value for r indicates a positive association, and a negative value for r indicates a negative association. The closer r is to 1 the closer the data points fall to a straight line, thus, the linear association is stronger. The closer r is to 0, making the linear association weaker.

## Correlation

Correlation is the statistical measure that defines to which extent two variables are linearly related to each other.  In statistics, correlation is defined by the Pearson Correlation formula : where,

• r: Correlation coefficient
• : i^th value first dataset X
• : Mean of first dataset X
• : i^th value second dataset Y
• : Mean of second dataset Y

Condition: The length of the dataset X and Y must be the same.

The  Correlation value can be positive, negative, or zeros. Correlation

### Implementations with code:

Import the numpy library and define a custom dataset x and y of equal length:

## Python3

 # Import the numpy libraryimport numpy as np # Define the datasetx = np.array([1,3,5,7,8,9, 10, 15])y = np.array([10, 20, 30, 40, 50, 60, 70, 80])

## Python3

 def Pearson_correlation(X,Y):    if len(X)==len(Y):        Sum_xy = sum((X-X.mean())*(Y-Y.mean()))        Sum_x_squared = sum((X-X.mean())**2)        Sum_y_squared = sum((Y-Y.mean())**2)              corr = Sum_xy / np.sqrt(Sum_x_squared * Sum_y_squared)    return corr             print(Pearson_correlation(x,y))   print(Pearson_correlation(x,x))

Output:

0.974894414261588
1.0

The above output shows that the relationship between x and y is 0.974894414261588 and x and x is 1.0

We can also find the correlation by using the numpy corrcoef function.

## Python3

 print(np.corrcoef(x, y))

Output:

[[ 1.         -0.97489441]
[-0.97489441  1.        ]]

The above output shows the correlations between x&x, x&y, y&x, and y&y.

## Python3

 import pandas as pdfrom sklearn.datasets import load_diabetesimport seaborn as snsimport matplotlib.pyplot as plt

## Python3

 # Load the dataset with framedf = load_diabetes(as_frame=True)# conver into pandas dataframedf = df.frame# Print first 5 rowsdf.head()

Output:

#### Find the Pearson correlations matrix by using the pandas command df.corr()

Syntax

df.corr(method, min_periods,numeric_only )

method : In method we can choose any one from {'pearson', 'kendall', 'spearman'} pearson is the standard correlation coefficient matrix i.e default
min_periods : int This is optional. Defines th eminimum number of observations required per pair.
numeric_only : Default is False, Defines we want to compare only numeric or categorical object also

## Python3

 # Find the pearson correlations matrixcorr = df.corr(method = 'pearson')corr

Output:

The above table represents the correlations between each column of the data frame. The correlation between the self is 1.0, The negative correlation defined negative relationship means on increasing one column value second will decrease and vice-versa. The zeros correlation defines no relationship I.e neutral. and positive correlations define positive relationships meaning on increasing one column value second will also increase and vice-versa.

We can also find the correlations using numpy between two columns

## Python3

 # correaltions between age and sex columnsc = np.corrcoef(df['age'],df['sex'])print('Correlations between age and sex\n',c)

Output:

Correlations between Age and Sex
[[1.        0.1737371]
[0.1737371 1.       ]]

We can match from the above correlation matrix table, it is almost the same result.

## Python3

 plt.figure(figsize=(10,8), dpi =500)sns.heatmap(corr,annot=True,fmt=".2f", linewidth=.5)plt.show()

Output: Correlations Matrix

My Personal Notes arrow_drop_up