NumPy | Replace NaN values with average of columns
Last Updated :
09 Feb, 2024
Data visualization is one of the most important steps in machine learning and data analytics.
Cleaning and arranging data is done by different algorithms. Sometimes in data sets, we get NaN (not a number) values that are unusable for data visualization.
To solve this problem, one possible method is to replace NaN values with an average of columns.
Given below are a few methods to solve this problem.
- Using np.colmean and np.take
- Using np.ma and np.where
- Using Naive and zip
- Using list comprehension and built-in functions
- Using zip()+lambda()
Let us understand them better with Python program examples:
Using np.colmean and np.take
We use the colmean() method of the NumPy library to find the mean of columns. We then use the take() method to replace column mean (average) with NaN values.
Example:
Python3
import numpy as np
ini_array = np.array([[ 1.3 , 2.5 , 3.6 , np.nan],
[ 2.6 , 3.3 , np.nan, 5.5 ],
[ 2.1 , 3.2 , 5.4 , 6.5 ]])
print ( "initial array" , ini_array)
col_mean = np.nanmean(ini_array, axis = 0 )
print ( "columns mean" , str (col_mean))
inds = np.where(np.isnan(ini_array))
ini_array[inds] = np.take(col_mean, inds[ 1 ])
print ( "final array" , ini_array)
|
Output:
initial array [[ 1.3 2.5 3.6 nan]
[ 2.6 3.3 nan 5.5]
[ 2.1 3.2 5.4 6.5]]
columns mean [ 2. 3. 4.5 6. ]
final array [[ 1.3 2.5 3.6 6. ]
[ 2.6 3.3 4.5 5.5]
[ 2.1 3.2 5.4 6.5]]
Using np.ma and np.where
We use the ma() method, which allows you to create a masked array where NaN values are masked out. We then use the where() method to replace the NaN values with column averages.
Example:
Python3
import numpy as np
ini_array = np.array([[ 1.3 , 2.5 , 3.6 , np.nan],
[ 2.6 , 3.3 , np.nan, 5.5 ],
[ 2.1 , 3.2 , 5.4 , 6.5 ]])
print ( "initial array" , ini_array)
res = np.where(np.isnan(ini_array), np.ma.array(ini_array,
mask = np.isnan(ini_array)).mean(axis = 0 ), ini_array)
print ( "final array" , res)
|
Output:
initial array [[ 1.3 2.5 3.6 nan]
[ 2.6 3.3 nan 5.5]
[ 2.1 3.2 5.4 6.5]]
final array [[ 1.3 2.5 3.6 6. ]
[ 2.6 3.3 4.5 5.5]
[ 2.1 3.2 5.4 6.5]]
Using Naive and zip
We use Zip to pair up the elements from the unpacked arrays, effectively giving us pairs of (row, column) indices for each NaN value in the array. We then replace these values with column averages.
Example:
Python3
import numpy as np
ini_array = np.array([[ 1.3 , 2.5 , 3.6 , np.nan],
[ 2.6 , 3.3 , np.nan, 5.5 ],
[ 2.1 , 3.2 , 5.4 , 6.5 ]])
print ( "initial array" , ini_array)
indices = np.where(np.isnan(ini_array))
for row, col in zip ( * indices):
ini_array[row, col] = np.mean(ini_array[
~np.isnan(ini_array[:, col]), col])
print ( "final array" , ini_array)
|
Output:
initial array [[ 1.3 2.5 3.6 nan]
[ 2.6 3.3 nan 5.5]
[ 2.1 3.2 5.4 6.5]]
final array [[ 1.3 2.5 3.6 6. ]
[ 2.6 3.3 4.5 5.5]
[ 2.1 3.2 5.4 6.5]]
Using list comprehension and built-in functions
It first computes the column means using a list comprehension with the help of the filter and zip functions. Then, it replaces the NaN values in the array with the corresponding column means using another list comprehension with the help of the enumerate function. Finally, it returns the modified list.
Algorithm:
1. Compute the column means.
2. Replace the NaN values in the array with the corresponding column means using list comprehension and built-in functions.
3. Return the modified list.
Python3
def replace_nan_with_mean(arr):
col_means = [ sum ( filter ( lambda x: x is not None , col)) / len ( list ( filter ( lambda x: x is not None , col))) for col in zip ( * arr)]
for i in range ( len (arr)):
arr[i] = [col_means[j] if x is None else x for j, x in enumerate (arr[i])]
return arr
arr = [[ 1.3 , 2.5 , 3.6 , None ],
[ 2.6 , 3.3 , None , 5.5 ],
[ 2.1 , 3.2 , 5.4 , 6.5 ]]
print (replace_nan_with_mean(arr))
|
Output
[[1.3, 2.5, 3.6, 6.0], [2.6, 3.3, 4.5, 5.5], [2.1, 3.2, 5.4, 6.5]]
Using zip()+lambda()
Compute the column means excluding NaN values using a loop over the transposed array zip(*arr). Replace NaN values with column means using map() and lambda functions.
Algorithm
1. Initialize an empty list means to store the column means.
2. Loop over the transposed array zip(*arr) to iterate over columns.
3. For each column, filter out None values and compute the mean of the remaining values. If there are no remaining values, set the mean to 0.
4. Append the mean to the means list.
5. Use map() and lambda functions to replace None values with the corresponding column mean in each row of the array arr.
6. Return the modified array arr.
Python3
arr = [[ 1.3 , 2.5 , 3.6 , None ],
[ 2.6 , 3.3 , None , 5.5 ],
[ 2.1 , 3.2 , 5.4 , 6.5 ]]
means = []
for col in zip ( * arr):
values = [x for x in col if x is not None ]
means.append( sum (values) / len (values) if values else 0 )
arr = list ( map ( lambda row: [means[j] if x is None else x for j,x in enumerate (row)], arr))
print (arr)
|
Output
[[1.3, 2.5, 3.6, 6.0], [2.6, 3.3, 4.5, 5.5], [2.1, 3.2, 5.4, 6.5]]
Share your thoughts in the comments
Please Login to comment...