How to Order Pyspark dataframe by list of columns ?
In this article, we are going to apply OrderBy with multiple columns over pyspark dataframe in Python. Ordering the rows means arranging the rows in ascending or descending order.
Method 1: Using OrderBy()
OrderBy() function is used to sort an object by its index value.
Syntax: dataframe.orderBy([‘column1′,’column2′,’column n’], ascending=True).show()
where,
- dataframe is the dataframe name created from the nested lists using pyspark
- where columns are the list of columns
- ascending=True specifies order the dataframe in increasing order, ascending=Falsespecifies order the dataframe in decreasing order
- show() method id used to display the columns.
Let’s create a sample dataframe
Python3
import pyspark
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName( 'sparkdf' ).getOrCreate()
data = [[ "1" , "sravan" , "vignan" ], [ "2" , "ojaswi" , "vvit" ],
[ "3" , "rohith" , "vvit" ], [ "4" , "sridevi" , "vignan" ],
[ "1" , "sravan" , "vignan" ], [ "5" , "gnanesh" , "iit" ]]
columns = [ 'student ID' , 'student NAME' , 'college' ]
dataframe = spark.createDataFrame(data, columns)
print ( "Actual data in dataframe" )
dataframe.show()
|
Output:
Applying OrderBy with multiple columns
Python3
dataframe.orderBy([ 'student ID' , 'student NAME' ],
ascending = True ).show()
|
Output:
Python3
dataframe.orderBy([ 'student ID' , 'student NAME' ],
ascending = False ).show()
|
Output:
Method 2: Using sort()
It takes the Boolean value as an argument to sort in ascending or descending order.
Syntax: dataframe.sort([‘column1′,’column2′,’column n’],ascending=True).show()
where,
- dataframe is the dataframe name created from the nested lists using pyspark
- where columns are the llst of columns
- ascending=True specifies order the dataframe in increasing order,ascending=Falsespecifies order the dataframe in decreasing order
- show() method id used to display the columns.
Python3
dataframe.sort([ 'college' , 'student NAME' ], ascending = False ).show()
|
Output:
Last Updated :
07 Nov, 2022
Like Article
Save Article
Share your thoughts in the comments
Please Login to comment...