Open In App

Exploring Correlation in Python

Improve
Improve
Improve
Like Article
Like
Save Article
Save
Share
Report issue
Report

This article aims to give a better understanding of a very important technique of multivariate exploration. A correlation Matrix is basically a covariance matrix. Also known as the auto-covariance matrix, dispersion matrix, variance matrix, or variance-covariance matrix. It is a matrix in which the i-j position defines the correlation between the ith and jth parameter of the given data set. When the data points follow a roughly straight-line trend, the variables are said to have an approximately linear relationship. In some cases, the data points fall close to a straight line, but more often there is quite a bit of variability of the points around the straight-line trend. A summary measure called correlation describes the strength of the linear association.

Correlation in Python

Correlation summarizes the strength and direction of the linear (straight-line) association between two quantitative variables. Denoted by r, it takes values between -1 and +1. A positive value for r indicates a positive association, and a negative value for r indicates a negative association. The closer r is to 1 the closer the data points fall to a straight line, thus, the linear association is stronger. The closer r is to 0, making the linear association weaker. 

Correlation

Correlation is the statistical measure that defines to which extent two variables are linearly related to each other.  In statistics, correlation is defined by the Pearson Correlation formula :

r = \frac{\sum(x_i -\bar{x})(y_i -\bar{y})}{\sqrt{\sum(x_i -\bar{x})^{2}\sum(y_i -\bar{y})^{2}}}

where,

  • r: Correlation coefficient 
  • x_i     : i^th value first dataset X
  • \bar{x}     : Mean of first dataset X
  • y_i     : i^th value second dataset Y
  • \bar{y}     : Mean of second dataset Y

Condition: The length of the dataset X and Y must be the same.

The  Correlation value can be positive, negative, or zeros.

Correlation

Correlation

Implementations with code:

Import the numpy library and define a custom dataset x and y of equal length:

Python3

# Import the numpy library
import numpy as np
 
# Define the dataset
x = np.array([1,3,5,7,8,9, 10, 15])
y = np.array([10, 20, 30, 40, 50, 60, 70, 80])

                    

Define the correlation by applying the above formula:

Python3

def Pearson_correlation(X,Y):
    if len(X)==len(Y):
        Sum_xy = sum((X-X.mean())*(Y-Y.mean()))
        Sum_x_squared = sum((X-X.mean())**2)
        Sum_y_squared = sum((Y-Y.mean())**2)      
        corr = Sum_xy / np.sqrt(Sum_x_squared * Sum_y_squared)
    return corr
             
print(Pearson_correlation(x,y))   
print(Pearson_correlation(x,x))

                    

Output:

0.974894414261588
1.0

The above output shows that the relationship between x and y is 0.974894414261588 and x and x is 1.0

We can also find the correlation by using the numpy corrcoef function.

Python3

print(np.corrcoef(x, y))

                    

Output:

[[ 1.         -0.97489441]
 [-0.97489441  1.        ]]

The above output shows the correlations between x&x, x&y, y&x, and y&y. 

Example

Import the necessary libraries 

Python3

import pandas as pd
from sklearn.datasets import load_diabetes
import seaborn as sns
import matplotlib.pyplot as plt

                    

Loading load_diabetes Data from sklearn.dataset

Python3

# Load the dataset with frame
df = load_diabetes(as_frame=True)
# conver into pandas dataframe
df = df.frame
# Print first 5 rows
df.head()

                    

Output: 


age
sexbmibps1s2s3s4s5s6target
00.0380760.0506800.0616960.021872-0.044223-0.034821-0.043401-0.0025920.019907-0.017646
1-0.001882-0.044642-0.051474-0.026328-0.008449-0.0191630.074412-0.039493-0.068332-0.092204
20.0852990.0506800.044451-0.005670-0.045599-0.034194-0.032356-0.0025920.002861-0.025930
3-0.089063-0.044642-0.011595-0.0366560.0121910.024991-0.0360380.0343090.022688-0.009362
40.005383-0.044642-0.0363850.0218720.0039350.0155960.008142-0.002592-0.031988-0.046641

Find the Pearson correlations matrix by using the pandas command df.corr()

Syntax 

df.corr(method, min_periods,numeric_only )

method : In method we can choose any one from {'pearson', 'kendall', 'spearman'} pearson is the standard correlation coefficient matrix i.e default
min_periods : int This is optional. Defines th eminimum number of observations required per pair.
numeric_only : Default is False, Defines we want to compare only numeric or categorical object also

Python3

# Find the pearson correlations matrix
corr = df.corr(method = 'pearson')
corr

                    

Output: 

 agesexbmibps1s2s3s4s5s6target
age1.0000000.1737370.1850850.3354280.2600610.219243-0.0751810.2038410.2707740.3017310.187889
sex0.1737371.0000000.0881610.2410100.0352770.142637-0.3790900.3321150.1499160.2081330.043062
bmi0.1850850.0881611.0000000.3954110.2497770.261170-0.3668110.4138070.4461570.3886800.586450
bp0.3354280.2410100.3954111.0000000.2424640.185548-0.1787620.2576500.3934800.3904300.441482
s10.2600610.0352770.2497770.2424641.0000000.8966630.0515190.5422070.5155030.3257170.212022
s20.2192430.1426370.2611700.1855480.8966631.000000-0.1964550.6598170.3183570.2906000.174054
s3-0.075181-0.379090-0.366811-0.1787620.051519-0.1964551.000000-0.738493-0.398577-0.273697-0.394789
s40.2038410.3321150.4138070.2576500.5422070.659817-0.7384931.0000000.6178590.4172120.430453
s50.2707740.1499160.4461570.3934800.5155030.318357-0.3985770.6178591.0000000.4646690.565883
s60.3017310.2081330.3886800.3904300.3257170.290600-0.2736970.4172120.4646691.0000000.382483
target0.1878890.0430620.5864500.4414820.2120220.174054-0.3947890.4304530.5658830.3824831.000000

The above table represents the correlations between each column of the data frame. The correlation between the self is 1.0, The negative correlation defined negative relationship means on increasing one column value second will decrease and vice-versa. The zeros correlation defines no relationship I.e neutral. and positive correlations define positive relationships meaning on increasing one column value second will also increase and vice-versa.

We can also find the correlations using numpy between two columns

Python3

# correaltions between age and sex columns
c = np.corrcoef(df['age'],df['sex'])
print('Correlations between age and sex\n',c)

                    

Output:

Correlations between Age and Sex
 [[1.        0.1737371]
 [0.1737371 1.       ]]

We can match from the above correlation matrix table, it is almost the same result.

Plot the correlation matrix with the seaborn heatmap

Python3

plt.figure(figsize=(10,8), dpi =500)
sns.heatmap(corr,annot=True,fmt=".2f", linewidth=.5)
plt.show()

                    

Output: 

Correlations Matrix - Geeksforgeeks

Correlations Matrix



Last Updated : 16 Mar, 2023
Like Article
Save Article
Previous
Next
Share your thoughts in the comments
Similar Reads