The use of real-world data in machine learning tasks can sometimes present challenges due to missing values or incomplete datasets. Such data can lead to inaccurate predictions, and ignoring the missing values can cause bias in model training and distort the original distribution of the data. As machine learning algorithms are not designed to handle missing data, it is important to either remove the missing values or fill the missing positions with other data. One way to fill in missing data values is through a process called Data Interpolation. In this blog, we will be thoroughly discussing about data interpolation.
What is Data Interpolation?
Data interpolation stands out as a crucial technique in data preprocessing, serving the purpose of estimating unknown values within the range of known data points. This method utilizes the existing data points to infer and fill in missing or unknown values in a dataset. The significance lies in its ability to replace missing values with predicted ones, enhancing the completeness and reliability of the dataset. In essence, data interpolation acts as a systematic tool, leveraging the available information to bridge gaps and provide a more comprehensive view of the data.
What data interpolation does is that it maps out an estimate between the known values such that the values present in between the known values that are absent could be replaced with the prediction through the estimated map. The assumption in the data interpolation process is that the changes between different points is continuous and smooth in nature. Through this assumption the data interpolation process is able to predict the unknown missing values.
Difference Between Interpolation and Extrapolation:
As data interpolation is a useful technique in estimating the data value in the range of the present data, data extrapolation helps in predicting the value outside of the range of data that is given to us. Interpolation and Extrapolation are sort of similar techniques which are used to handle data, both the processes serve different purposes have different merits of their use and different demerits. Here we will be discussing the key differences between these two processes.
| Interpolation
| Extrapolation
|
|---|
Defination
| Interpolation predicts the values which are present within the known data range.
| Extrapolation estimates the values which are present outside of the known data range.
|
|---|
Uncertainity
| Interpolation is considered to be a more reliable process as it depends on the observed data
| Extrapolation involves high risk, it assumes that the observed trends in the data continues outside of the known data range.
|
|---|
Dependence on Data
| Interpolation directly depends on the known data points.
| Extrapolation depends on the assumption that the trends in the known data continues outside the data range.
|
|---|
Accuracy
| Interpolation is often more accurate than extrapolation since it is based on the existing data.
| Extrapolation is less accurate in nature as it assumes the data trends to continue for values out of range.
|
|---|
Example
| Measurement of the atmospheric pressure is present at 1km above the ground and 2km above the ground and atmospheric pressure at 1.5 km from the ground is required.
| Measurement of the atmospheric pressure is given at 1km above the ground and 2km above the ground and atmospheric pressure at 3 km from the ground is required.
|
|---|
Need of Data Interpolation
Data Interpolation have different uses in applications related to data science, especially in the field of data analysis and scientific research. It can help in enhancing our understanding of the data which is present, and it can elaborate the trends seen in the data in a better manner, let’s see some of the most important reasons why data interpolation is performed:
- Missing Data Handling: Data Interpolation helps in handling missing data points in the dataset. This is an essential step to be followed in order to fill up the gap in the data which occurs due measurement errors, malfunctioning equipment or some other reason.
- Smoothing Noise Data: Data Interpolation helps in smoothing out the dataset by reducing the impact of random fluctuations and create a clear-cut representation of underlying trends in the data. The sudden spikes in the dataset could be removed with the help of data interpolation, we use the interpolation method to interpolate the point where the spike in the data is present.
- Creating Continuous Dataset: Continuous distribution of dataset could be created with the help of data interpolation; the specified data interpolation method could be used to find out the data point intermediate data points in between the given data points such that a continuous dataset is created.
- Enhancing Visualization: As we have seen data interpolation provides a smoother and continuous dataset which becomes easy to visualize and understand, the reduction of noise and the enhancement of trends in the dataset increases the aesthetics of the visualizations, ultimately increasing the ability of visualizations to extract useful information.
Types of Data Interpolation
There are many types of interpolation methods which could be used as per the nature of data present and the available computational resources, let’s discuss about some of the most important interpolation methods that can be useful to us in our data science projects:
Linear Interpolation:
Linear Interpolation is a data interpolation technique in which the relationship in between different data points is taken to be linear in nature and the value of unknown data points is estimated through the linear plot in between known points. For example two of the data points are given which have the coordinates (
) and (
) then a straight line is plotted in between these data points as 
and the data points are estimated according to this equation. Linear interpolation is good if the relationship between different variables are linear in nature.
Polynomial Interpolation:
Polynomial Interpolation on the other hand uses polynomial equation to find out missing data values. It passes through all the data points present in the dataset and the degree of the polynomial is one less than the total number of data points present in the dataset. Polynomial interpolation can fit complex datasets to it but if the dataset is really large this might lead to the interpolation method to overfit the dataset and the estimates of the unknown value gets poor in nature. Still polynomial interpolation method is way more flexible than linear interpolation and it could be used for complex datasets. There are different types of polynomial interpolation methods, let’s discuss about some of the most commonly used polynomial interpolation methods:
- Lagrange Interpolation: Lagrange’s Interpolation method creates a polynomial which passes through all the data points which are given. For a total no n given data points this method creates a polynomial of n-1 degree. As it considers all the given data points, it is considered to be computionally expensive.
- Newton Interpolation: Newton’s interpolation develops a polynomial function using the concept of divided difference. Newton’s method builds up the polynomial by adding terms that consider an increasing number of points. It is computationally better than the lagrange’s interpolation.
- Hermite Interpolation: Hermite Interpolation extends the idea of polynomial interpolation by considering the derivatives at the points of given data with the given data. It is considered to be more complex as it is also considering slope at each points.
Spline Interpolation:
In Spline interpolation the data set is divided into small chunks of data on which the low degree polynomials are applied. This method reduces the risk of overfitting that comes with polynomial interpolation as lower degree polynomial equations are used instead of higher degree polynomial equations. Spline interpolation produces smoother results than polynomial interpolation, majorly third degree polynomials are used for the interpolation method to estimate missing unknown values.
Nearest Neighbor Interpolation:
The Nearest Neighbor Interpolation is the data interpolation technique in which the unknown data is assigned some value based on it’s nearest neighbor. This method is considered one of the most important interpolation method which is used for image processing, it assumes that the nearest unknown point to the known data point must have similar characteristics as of the known data point. The major drawback that is seen in this method is that there might be a possibility of the so called ‘staircase effect‘ which produces less smooth transition between pixels, since nearest neighbor interpolation only considers the nearest neighbor and not other neighbors surrounding the missing value.
Application of Data Interpolation with Code Example
Here we will be exploring some of the most common applications of data interpolation with their code examples:
1. Time Series Analysis:
If the time series is irregularly sampled it becomes really a mess to analyze such time series, as the data of the time series is continuously monitored with respect to time, data interpolation plays a crucial role in determining the missing values of the time series. Assuming there is no anamoly in place of the missing data, data interpolation interpolates the missing values with the method of interpolation chosen. Let’s see an example for the same,
First we will be importing numpy library which will be useful in scientific mathematical operations and manipulation of numeric data, after that we will be creating our own time series with synthetic data points and a value to be equal to not a number, which is missing value in our dataset.
Python3
import numpy as np
import pandas as pd
dates = pd.to_datetime(['2022-01-01', '2022-01-02', '2022-01-03', '2022-01-04', '2022-01-05'])
values = np.array([10, 15, 20, np.nan, 30])
|
After creating synthetic dataset we will be interpolating the original dataset with the value received from linear interpolation of the given data points. ‘np.interp‘ is a linear interpolation method in which there are three parameters defined.
First parameter time_points is the x-coordinates of the data where the interpolation is supposed to be calculated. The second parameter time_points[~np.isnan(values)] defines the values of the x-coordinates where the value of the data is not missing and the third coordinate values[~np.isnan(values)] is the values of the data which is not missing. This method will map out the give data and fill up the missing data with the interpolation map which is created through the given data.
Python3
interpolated_values = np.interp(dates, dates[~np.isnan(values)], values[~np.isnan(values)])
|
In the last stage of the code we will be filling the missing value with the interpolated value and printing the original dataset with the interpolated dataset, such that we can see that the missing value is correctly interpolated.
Python3
interpolated_data = pd.DataFrame({
'Date': dates,
'Original_Value': values,
'Interpolated_Value': interpolated_values
})
print(interpolated_data)
|
Output:
Date Original_Value Interpolated_Value
0 2022-01-01 10.0 10.0
1 2022-01-02 15.0 15.0
2 2022-01-03 20.0 20.0
3 2022-01-04 NaN 25.0
4 2022-01-05 30.0 30.0
2. Image Processing:
Data interpolation also have application in the field of image processing, it helps in resizing the image and the enhancement of the image, in this example we will be using data interpolation for the purpose of image processing. First in this process we will be importing the important libraries for processing the image, this include ndimage from scipy library for image processing operations, matplotlib.pyplot for the visualization purpose, and data from skimage to access sample image.
Python3
from scipy import ndimage
import matplotlib.pyplot as plt
from skimage import data
|
After importing all the required libraries, we will be loading an image and resizing it such that bilinear interpolation could be applied to it. The data.camera() method from the skimage library is used to load a sample image, the camera camera provides a sample image of a camera. Here ndimage.zoom is used to resize the loaded image, The zoom parameter is set to 2 to specify the scaling factor, The order parameter is set to 1 to specify bilinear interpolation.
Python3
image = data.camera()
resized_image = ndimage.zoom(image, zoom=2, order=1)
|
After resizing the image we will be visualizing and comparing both the original and the resized image. We will be creating a plot with two subplots which represents each original and resized image.
Python3
plt.figure(figsize=(8, 4))
plt.subplot(1, 2, 1)
plt.title("Original Image")
plt.imshow(image, cmap='gray')
plt.subplot(1, 2, 2)
plt.title("Resized Image")
plt.imshow(resized_image, cmap='gray')
plt.show()
|
Output:
Original vs Resized Interpolated Image
Here we have discussed two of the main application with the code displaying data interpolation, it has many more applications associated with it which could be related with the basic purpose of data interpolation.
Tools and Software for Data Interpolation:
As we have talked about data interpolation in detail, to perform data interpolation there are several tools and software packages available, ranging from general-purpose programming languages to specialized tools designed for specific fields. Here are some of the tools and software commonly used for data interpolation:
- Python: As used in the above examples Python programming language is one of the most commonly tools used for data interpolation, with data centric libraries like numpy, pandas, and scipy, it gets pretty easy to interpolate data through Python.
- R: R is another data centric programming language used specially for the purpose of data science, it comprises of different packages which could be used for the purpose of data interpolation for example zoo and xts which are commonly used for the purpose of time series analysis could help in data interpolation and package like akima could be used for the purpose of polynomial interpolation.
- MATLAB: Scientific Research language could be also of use for data interpolation, it contains built-in functions such as interp1 for one dimensional interpolation and griddata for grid based interpolation.
- Excel: Tools like Microsoft Excel also contains built-in functions like LINEST for data interpolation, that could be used for linear regression and basic interpolation purpose.
- SAS (Statistical Analysis System): SAS is most commonly used for the purpose of statistical data analysis, it contains different interpolation functions that could be used for numeric data interpolation.
Finally, the choice of tool or software depends on the specific requirements of the data which is given, the domain of application, and familiarity with the tools. The mentioned tools offer a combination of ease of use, versatility, and specialized functionalities for different interpolation scenarios.
Advantages and Disadvantages of Data Interpolation:
There are certain advantages of data interpolation but with the advantages we must also focus our attention on the disadvantages of data interpolation so that we can improve on them and work with the data interpolation process carefully which profits us in whole.
Advantages
- Fill Missing Data: Data interpolation helps in handling missing values in the dataset by fitting interpolated values in place of missing values.
- Spatial Analysis: Data interpolation is an important concept in spatial analysis, since it helps in the estimation of values in places where measurements weren’t taken.
- Algorithmic Efficiency: Data interpolation increases the efficiency of machine learning algorithms as it handles the missing data well it helps algorithms learn better and perform better in their prediction.
- Smoothing: Interpolation helps in smoothing out the values where measurements were not taken making the whole process continuous in nature.
Disadvantages
- Assumption of Smoothness: Interpolation methods assume that the data is smooth, if the data contains outliers or abrupt changes the interpolation methods might not be good to capture such features.
- Loss of Information: Interpolation uses the known data to estimate the value of the missing or unknown data, therefore, it assumes a smooth and continuous relationship between data which might result in loss of information.
- Extrapolation Uncertainty: Extrapolation deals with the unknown data outside of the range of the known data, but interpolation only deals with the data present inside the range of the known data. Thus we cannot use interpolation to extrapolate unknown data.
- Computational Intensity: Some of the interpolation methods might be computationally complex, which might slow down the process of interpolation. Therefore, we must choose the interpolation method with full consideration.
Conclusion
In this blog we have seen that how data interpolation can help us replace unknown missing value with a fairly better value which matches the dataset present to us. But we must be careful in choosing the interpolation method, since it determine, what values are going to fit in to our dataset.
Share your thoughts in the comments
Please Login to comment...