Skip to content
Related Articles

Related Articles

Improve Article

Show distinct column values in PySpark dataframe

  • Last Updated : 06 Jun, 2021

In this article, we are going to display the distinct column values from dataframe using pyspark in Python. For this, we are using distinct() and dropDuplicates() functions along with select() function.

Let’s create a sample dataframe.

Python3




# importing module
import pyspark
  
# importing sparksession from pyspark.sql module
from pyspark.sql import SparkSession
  
# creating sparksession and giving an app name
spark = SparkSession.builder.appName('sparkdf').getOrCreate()
  
# list  of employee data with 5 row values
data =[["1", "sravan", "company 1"],
       ["3", "bobby", "company 3"],
       ["2", "ojaswi", "company 2"],
       ["1", "sravan", "company 1"],
       ["3", "bobby", "company 3"],
       ["4", "rohith", "company 2"],
       ["5", "gnanesh", "company 1"]]
  
# specify column names
columns = ['Employee ID','Employee NAME','Company Name']
  
# creating a dataframe from the lists of data
dataframe = spark.createDataFrame(data,columns)
  
dataframe.show()

Output:



Method 1: Using distinct()

This function returns distinct values from column using distinct() function.

Syntax: dataframe.select(“column_name”).distinct().show()

Example1: For a single column.

Python3




# select first column to get
# unique data using distinct function()
dataframe.select("Employee ID").distinct().show()

Output:

+-----------+
|Employee ID|
+-----------+
|          3|
|          5|
|          1|
|          4|
|          2|
+-----------+

Example 2: For multiple columns.

Python code to display unique data from 2 columns using distinct() function.

Syntax: dataframe.select(“column_name 1, column_name 2 “).distinct().show()



Code:

Python3




# select first and second column
# to get unique data using distinct function()
dataframe.select(["Employee ID",
                  "Employee NAME"]).distinct().show()

Output:

+-----------+-------------+
|Employee ID|Employee NAME|
+-----------+-------------+
|          5|      gnanesh|
|          4|       rohith|
|          1|       sravan|
|          2|       ojaswi|
|          3|        bobby|
+-----------+-------------+

Method 2: Using dropDuplicates()

This function displays unique data in one column from dataframe using dropDuplicates() function.

Syntax: dataframe.select(“column_name”).dropDuplicates().show()

Example 1: For single columns.

Python3




#select first  column to get
# unique data using dropDuplicates function()
dataframe.select("Employee ID").dropDuplicates().show()

Output:

+-----------+
|Employee ID|
+-----------+
|          3|
|          5|
|          1|
|          4|
|          2|
+-----------+

Example 2: For multiple columns

Python code to display unique data from 2 columns using dropDuplicates() function

Python3




#select first and second column
# to get unique data using dropDuplicates function()
dataframe.select(["Employee ID",
                  "Employee NAME"]).dropDuplicates().show()

Output:

+-----------+-------------+
|Employee ID|Employee NAME|
+-----------+-------------+
|          5|      gnanesh|
|          4|       rohith|
|          1|       sravan|
|          2|       ojaswi|
|          3|        bobby|
+-----------+-------------+

 Attention geek! Strengthen your foundations with the Python Programming Foundation Course and learn the basics.  

To begin with, your interview preparations Enhance your Data Structures concepts with the Python DS Course. And to begin with your Machine Learning Journey, join the Machine Learning – Basic Level Course




My Personal Notes arrow_drop_up
Recommended Articles
Page :