PySpark – Select Columns From DataFrame
Last Updated :
04 Aug, 2021
In this article, we will discuss how to select columns from the pyspark dataframe. To do this we will use the select() function.
Syntax: dataframe.select(parameter).show()
where,
- dataframe is the dataframe name
- parameter is the column(s) to be selected
- show() function is used to display the selected column
Let’s create a sample dataframe
Python3
import pyspark
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName( 'sparkdf' ).getOrCreate()
data = [[ "1" , "sravan" , "vignan" ], [ "2" , "ojaswi" , "vvit" ],
[ "3" , "rohith" , "vvit" ], [ "4" , "sridevi" , "vignan" ],
[ "1" , "sravan" , "vignan" ], [ "5" , "gnanesh" , "iit" ]]
columns = [ 'student ID' , 'student NAME' , 'college' ]
dataframe = spark.createDataFrame(data, columns)
print ( "Actual data in dataframe" )
dataframe.show()
|
Output:
Selecting single Column
With column name, we can get the whole column in the data frame
Syntax: dataframe.select(“column_name”).show()
Python3
dataframe.select( 'student ID' ).show()
|
Output:
Selecting multiple Columns
With multiple column names, we can get the whole column in the data frame
Syntax: dataframe.select([“column_name1″,”column_name 2″,”column_name n”]).show()
Python3
dataframe.select([ 'student ID' , 'student NAME' , 'college' ]).show()
|
Output:
Selecting using Column Number
Here we are going to select the columns based on the column number. This can be done using the indexing operator. We can pass the column number as the index to dataframe.columns[].
Syntax: dataframe.select(dataframe.columns[column_number]).show()
Python3
dataframe.select(dataframe.columns[ 1 ]).show()
|
Output:
Accessing multiple columns based on column number. Here we are going to select multiple columns by using the slice operator.
Syntax: dataframe.select(dataframe.columns[column_start:column_end]).show()
where, column_start is the starting index and column_end is the ending index
Python3
dataframe.select(dataframe.columns[ 0 : 3 ]).show()
|
Output:
Share your thoughts in the comments
Please Login to comment...