How to select last row and access PySpark dataframe by index ?
In this article, we will discuss how to select the last row and access pyspark dataframe by index.
Creating dataframe for demonstration:
Select last row from dataframe
Example 1: Using tail() function.
This function is used to access the last row of the dataframe
- n is the number of rows to be selected from the last.
- dataframe is the input dataframe
We can use n = 1 to select only last row.
Example 1: Selecting last row.
[Row(student ID=’5′, student NAME=’gnanesh’, college=’iit’)]
Example 2: Python program to access last N rows.
[Row(student ID='2', student NAME='ojaswi', college='vvit'), Row(student ID='3', student NAME='rohith', college='vvit'), Row(student ID='4', student NAME='sridevi', college='vignan'), Row(student ID='1', student NAME='sravan', college='vignan'), Row(student ID='5', student NAME='gnanesh', college='iit')]
Access the dataframe by column index
Here we are going to select the dataframe based on the column number. For selecting a specific column by using column number in the pyspark dataframe, we are using select() function
- dataframe is the dataframe name
- dataframe.columns: is the method which can take column number as an input and select those column
- show() function is used to display the selected column
Example 1: Python program to access column based on column number
+------------+ |student NAME| +------------+ | sravan| | ojaswi| | rohith| | sridevi| | sravan| | gnanesh| +------------+
Example 2: Accessing multiple columns based on column number, here we are going to select multiple columns by using the slice operator, It can access upto n columns
where: column_start is the starting index and column_end is the ending index.
Attention geek! Strengthen your foundations with the Python Programming Foundation Course and learn the basics.
To begin with, your interview preparations Enhance your Data Structures concepts with the Python DS Course. And to begin with your Machine Learning Journey, join the Machine Learning – Basic Level Course