Winsorization

Winsorization is the process of replacing the extreme values of statistical data in order to limit the effect of the outliers on the calculations or the results obtained by using that data. The mean value calculated after such replacement of the extreme values is called winsorized mean.

 For example, 90% winsorization means the replacement of the top 5% and bottom 5% of the data. The top 5% of the data is replaced by the value of the data at the 95th percentile and the value of the bottom 5% of the data is replaced by the value of the data at the 5th percentile. 

Input:

• A numeric array whose values at the upper end and the lower end are to be winsorized.
• The first argument of the tuple is the percentage of values at the lower end which are to be winsorized.
• The second argument of the tuple is the percentage of values at the upper end which are to be winsorized.

Output:

A numeric array whose values at the upper end and at the lower end are winsorized as defined by the user.

Example #1:

Python3

# Libraries to be imported
import numpy as np
import matplotlib.pyplot as plt
import random
from scipy.stats.mstats import winsorize

Let us see an example where outliers are present on both the upper end and the lower end of the data.

Python3

# Creating an array with 100 random values
array = [np.random.randint(100) for i in range(100)]

# Creating outliers
# Here, the values which are selected for creating outliers
# are appended so that same outliers are not created again.
AlreadySelected = []
i = 0

# Creating 5 outliers on the lower end
while (i < 5):
    x = np.random.choice(array)  # Randomly selecting a value from the array
    y = x - mean*3
    array = np.append(array, y)
    if (x not in already_selected):
        AlreadySelected.append(y)

        i += 1

    else:
        continue

# Creating 5 outliers on the upper end
i = 0
while (i < 5):
    x = np.random.choice(array)  # Randomly selecting a value from the array
    y = x + mean*4
    array = np.append(array, y)
    if (x not in already_selected):
        AlreadySelected.append(y)

        i += 1

    else:
        continue

std = np.std(array)  # Storing the standard deviation of the array
mean = np.mean(array)  # Storing the mean of the array

plt.boxplot(array)
plt.title('Array with Outliers')
plt.show()

Output:

Python3

print(mean) # mean of the numeric array with outliers

Output:

Now, we winsorize the array by 10% i.e. we winsorize 5% of the highest values and 5% of the lowest value of the array:

Python3

WinsorizedArray = winsorize(array,(0.05,0.05))

plt.boxplot(WinsorizedArray)
plt.title('Winsorized array')
plt.show()

Output:

Python3

WinsorizedMean = np.mean(WinsorizedArray)
print(WinsorizedMean)

Output:

In this case, there is only a slight change in the mean value of the data.

Now, let us see an example where outliers are present only at one end of the data.

Python3

# Creating another array with 100 random values
array2 = [np.random.randint(100) for i in range(100)]
std = np.std(array2)
mean = np.mean(array2)
AlreadySelected = []
# Creating outliers on the upper end
i = 0
while (i<5):
    x = np.random.choice(array2) # Randomly selecting a value from the array
    y = x + mean*4
    array2 = np.append(array2,y)
    if (x not in AlreadySelected):
        AlreadySelected.append(y)

        i+=1

    else:
        continue

plt.boxplot(array2)
plt.title('Array with outliers')
plt.show()

Output:

Python3

OutlierArray2Mean = np.mean(array2)
print(OutlierArray2Mean)

Output:

Python3

WinsorizedArray2 = winsorize(array2,(0.1,0.1))
# In this case, the lower 10% values of
# the data will have their values set equal to the value of the data point at
#the 10th percentile.

plt.boxplot(WinsorizedArray2)
plt.show()

WinsorizedArray2Mean = np.mean(WinsorizedArray2)

Output:

Python3

WinsorizedArray2Mean = np.mean(WinsorizedArray2)
print(WinsorizedArray2Mean)

Output:

In this case, there is a significant difference in the mean value.