In the case of a real-world dataset, it is very common that some values in the dataset are missing. We represent these missing values as NaN (Not a Number) values. But to build a good machine learning model our dataset should be complete. That’s why we use some imputation techniques to replace the NaN values with some probable values. But before doing that we need to have a good understanding of how the NaN values are distributed in our dataset.
Missingno library offers a very nice way to visualize the distribution of NaN values. Missingno is a Python library and compatible with Pandas.
Install the library –
pip install missingno
To get the dataset used in the code, click here.
Using this matrix you can very quickly find the pattern of missingness in the dataset. In our example, the columns
SulphidityL-4 have a similar pattern of missing values while UCZAA shows a different pattern.
Bar Chart :
This bar chart gives you an idea about how many missing values are there in each column. In our example,
SulphidityL-4 contain the most number of missing values followed by UCZAA.
Heatmap shows the correlation of missingness between every 2 columns. In our example, the correlation between AAWhiteSt-4 and SulphidityL-4 is 1 which means if one of them is present then the other one must be present.
A value near -1 means if one variable appears then the other variable is very likely to be missing.
A value near 0 means there is no dependence between the occurrence of missing values of two variables.
A value near 1 means if one variable appears then the other variable is very likely to be present.
Reference : https://github.com/ResidentMario/missingno
- Python | Find missing and additional values in two lists
- ML | Handling Missing Values
- Mapping external values to dataframe values in Pandas
- Python dictionary | values()
- Python | Segregating Key's Values
- Python | Pandas DataFrame.values
- Python | Pandas Index.values
- Python | Check if all values are 0 in dictionary
- Python | Check if all values in numpy are zero
- Python | Get values of particular key in list of dictionaries
- Python | Check if all the values in a list are less than a given value
- Python | Initialize dictionary with None values
- Python | Selective key values in dictionary
- Returning Multiple Values in Python
- Python | Combine the values of two dictionaries having same key
If you like GeeksforGeeks and would like to contribute, you can also write an article using contribute.geeksforgeeks.org or mail your article to firstname.lastname@example.org. See your article appearing on the GeeksforGeeks main page and help other Geeks.
Please Improve this article if you find anything incorrect by clicking on the "Improve Article" button below.