In the field of data science and machine learning, a correlation matrix aids in understanding relationships between variables. Correlation matrix represents how different variables interact with each other.
For someone who is navigating the complex landscape of data, understanding and harnessing the potential of correlation matrices is a skill that can significantly enhance their ability to drive meaningful insights. In this article, we will explore the step-by-step process of creating a correlation matrix in Python.
What is correlation?
Correlation is a statistical indicator that quantifies the degree to which two variables change in relation to each other. It indicates the strength and direction of the linear relationship between two variables. The correlation coefficient is denoted by “r”, and it ranges from -1 to 1.
- If r = -1, it means that there is a perfect negative correlation.
- If r = 0, it means that there is no correlation between the two variables.
- If r = 1, it means that there is a perfect positive correlation.
There are two popular methods used to find the correlation coefficients:
Pearson’s product-moment correlation coefficient
The Pearson correlation coefficient (r) is a measure of linear relationship between two variables.
r = n(∑xy) – (∑x)(∑y) / √[n∑x²-(∑x)²][n∑y²-(∑y)²]
Here,
- n is the number of data points
- ∑xy is the sum of the product of corresponding values of x and y
- ∑x is the sum of all the values of x
- ∑y is the sum of all the values of y
- ∑x^2 is the sum of the squares of all values of x
- ∑y^2 is the sum of the squares of all the of y
Spearman’s rank correlation coefficient
The Spearman’s rank correlation coefficient is a measure of statistical dependence between two variables. It is based on the ranks of the data rather than the actual data values.
\rho = 1 – \frac{6 \sum d^2}{n(n^2 -1)}
Here,
- n is the number of paired observations
- d is the difference between the rank of corresponding values of the two variables.
What is a Correlation Matrix?
A correlation is a tabular representation that displays correlation coefficients, indicating the strength and direction of relationships between variables in a dataset. Within this matrix, each cell signifies the correlation between two specific variables. This tool serves multiple purposes, serving as a summary of data relationships, input for more sophisticated analyses, and a diagnostic aid for advanced analytical procedures. By presenting a comprehensive overview of inter-variable correlations, the matrix becomes invaluable in discerning patterns, guiding further analyses, and identifying potential areas of interest or concern in the dataset. Its applications extend beyond mere summary statistics, positioning it as a fundamental component in the preliminary stages of diverse and intricate data analyses.
Interpreting the correlation matrix
- Strong correlations, indicated by values close to 1 or -1, suggest a robust connection, while weak correlations, near 0, imply a less pronounced association. They are identifying these degrees of correlation aids in understanding the intensity of interactions within the dataset, facilitating targeted analysis and decision-making.
- Positive correlations (values > 0) signify that as one variable increases, the other tends to increase as well. Conversely, negative correlations (values < 0) imply an inverse relationship—when one variable increases, the other tends to decrease. Investigating these directional associations provides insights into how variables influence each other, crucial for formulating informed hypotheses and predictions.
How to create correlation matrix in Python?
A correlation matrix has been created using the following two libraries:
- NumPy Library
- Pandas Library
Creating a correlation matrix using NumPy Library
NumPy is a library for mathematical computations. It can be used for creating correlation matrices that helps to analyze the relationships between the variables through matric representation.
Example 1
Suppose an ice cream shop keeps track of total sales of ice creams versus the temperature on that day. To learn the correlation, we will use NumPy library.
In the following code snippet, x and y represent total sales in dollars and corresponding temperatures for each day of sale and np.corrcoef() function is sed to compute the correlation matrix.
import numpy as np
# x represents the total sale in dollars
x = [215, 325, 185, 332, 406, 522, 412,
614, 544, 421, 445, 408],
# y represents the temperature on each day of sale
y = [14.2, 16.4, 11.9, 15.2, 18.5, 22.1,
19.4, 25.1, 23.4, 18.1, 22.6, 17.2]
# create correlation matrix
matrix = np.corrcoef(x, y)
print(matrix)
Output:
[[1. 0.95750662]
[0.95750662 1. ]]
From the above matrix, if we see cell (0,1) and (1,0) both have the same value equal to 0.95750662 which lead us to conclude that whenever the temperature is high, we have more sales. Let’s have a look at another example.
Example 2
Suppose we are given glucose level in boy respective to age. To find correlation between age(x) and glucose level in body(y), we will again use NumPy library. In the following code snippet, the variables x and y represent age and corresponding glucose levels. The np.corrcoef() is used to calculate the correlation between the x and y variables.
import numpy as np
# x represents the age
x = [43, 21, 25, 42, 57, 59]
# y represents the glucose level corresponding to that age
y = [99, 65, 79, 75, 87, 81]
# correlation matrix
matrix = np.corrcoef(x, y)
print(matrix)
Output
[[1. 0.5298089]
[0.5298089 1. ]]
From the above correlation matrix, 0.5298089 or 52.98% that means the variable has a moderate positive correlation.
Creating correlation matrix using Pandas library
Pandas is a library with built-in functionalities using which user can analyze and interpret the relationships between variables.
Example 1:
To illustrate this example, we have created a data frame with three variables and have calculated the correlation matrix. In order to create a correlation matrix, we used corr() method on data frames.
import pandas as pd
# collect data
data = {
'x': [45, 37, 42, 35, 39],
'y': [38, 31, 26, 28, 33],
'z': [10, 15, 17, 21, 12]
}
# form dataframe
dataframe = pd.DataFrame(data, columns=['x', 'y', 'z'])
print("Dataframe is : ")
print(dataframe)
# form correlation matrix
matrix = dataframe.corr()
print("Correlation matrix is : ")
print(matrix)
Output:
Dataframe is :
x y z
0 45 38 10
1 37 31 15
2 42 26 17
3 35 28 21
4 39 33 12
Correlation matrix is :
x y z
x 1.000000 0.518457 -0.701886
y 0.518457 1.000000 -0.860941
z -0.701886 -0.860941 1.000000
Example 2:
In this example, we will consider Iris dataset and find correlation between the features of the dataset.
from sklearn import datasets
import pandas as pd
#load iris dataset
dataset = datasets. load_iris ()
dataframe = pd. DataFrame (data = dataset. data, columns = dataset. feature_names)
dataframe ["target"] = dataset. target
#correlation matrix
matrix = dataframe.corr()
print(matrix)Output:
sepal length (cm) sepal width (cm) petal length (cm) \
sepal length (cm) 1.000000 -0.117570 0.871754
sepal width (cm) -0.117570 1.000000 -0.428440
petal length (cm) 0.871754 -0.428440 1.000000
petal width (cm) 0.817941 -0.366126 0.962865
target 0.782561 -0.426658 0.949035
petal width (cm) target
sepal length (cm) 0.817941 0.782561
sepal width (cm) -0.366126 -0.426658
petal length (cm) 0.962865 0.949035
petal width (cm) 1.000000 0.956547
target 0.956547 1.000000
How to visualize correlation matrix in Python?
There are two popular libraries for data visualization, Matplotlib and Seaborn. Let’s first visualize the code using Matplotlib using the iris dataset.
Correlation matrix using Matplotlib
In the visualization part of the code,
- plt.imshow() function is used to create a heatmap.
- plt.colobar() is used to add the colorbar to the heat map.
- the name of the features of iris dataset are stored in the ‘variables’.
- plt.xticks() and plt.yticks() is used to add labels to the matrix.
Python3
import matplotlib.pyplot as plt
from sklearn import datasets
import pandas as pd
dataset = datasets. load_iris ()
dataframe = pd. DataFrame (data = dataset. data, columns = dataset. feature_names)
dataframe ["target"] = dataset. target
matrix = dataframe.corr()
#plotting correlation matrix
plt.imshow(matrix, cmap='Blues')
#adding colorbar
plt.colorbar()
#extracting variable names
variables = []
for i in matrix.columns:
variables.append(i)
# Adding labels to the matrix
plt.xticks(range(len(matrix)), variables, rotation=45, ha='right')
plt.yticks(range(len(matrix)), variables)
# Display the plot
plt.show()
Output:
Correlation Matrix using Seaborn
Let’s visualize using Seaborn.
Python3
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn import datasets
import pandas as pd
dataset = datasets. load_iris ()
dataframe = pd. DataFrame (data = dataset. data, columns = dataset. feature_names)
dataframe ["target"] = dataset. target
matrix = dataframe.corr()
#plotting correlation matrix
sns.heatmap(matrix, cmap="Greens", annot=True)Output:
Also Check:
- Correlation: Meaning, Significance, Types and Degree of Correlation
- Correlation Matrix in R Programming
- How to Create a Correlation Matrix using Pandas?
- Exploring Correlation in Python