Skip to content
Related Articles

Related Articles

Save Article
Improve Article
Save Article
Like Article

PySpark – Extracting single value from DataFrame

  • Last Updated : 17 Jun, 2021

In this article, we are going to extract a single value from the pyspark dataframe columns. To do this we will use the first() and head() functions.

Single value means only one value, we can extract this value based on the column name

 Attention geek! Strengthen your foundations with the Python Programming Foundation Course and learn the basics.  

To begin with, your interview preparations Enhance your Data Structures concepts with the Python DS Course. And to begin with your Machine Learning Journey, join the Machine Learning - Basic Level Course

Syntax



  • dataframe.first()[‘column name’]
  • Dataframe.head()[‘Index’]

Where,

  • dataframe is the input dataframe and column name is the specific column
  • Index is the row and columns.

So we are going to create the dataframe using the nested list.

Python3




# importing module
import pyspark
  
# importing sparksession from pyspark.sql module
from pyspark.sql import SparkSession
  
# creating sparksession and giving an app name
spark = SparkSession.builder.appName('sparkdf').getOrCreate()
  
# list  of students  data 
data =[["1","sravan","vignan"],
       ["2","ojaswi","vvit"],
       ["3","rohith","vvit"],
       ["4","sridevi","vignan"],
       ["1","sravan","vignan"], 
       ["5","gnanesh","iit"]]
  
# specify column names
columns=['student ID','student NAME','college']
  
# creating a dataframe from the lists of data
dataframe = spark.createDataFrame(data,columns)
  
print("Actual data in dataframe")
# show dataframe
dataframe.show()

Output:

Actual data in dataframe
+----------+------------+-------+
|student ID|student NAME|college|
+----------+------------+-------+
|         1|      sravan| vignan|
|         2|      ojaswi|   vvit|
|         3|      rohith|   vvit|
|         4|     sridevi| vignan|
|         1|      sravan| vignan|
|         5|     gnanesh|    iit|
+----------+------------+-------+

Example 1: Python program to extract a single value from a particular column using first().

Python3




# extract single value based on
# column in the dataframe
dataframe.first()['student ID']

Output:

'1'

Example 2: Extract a single value using head().

Python3




# extract single value based
# on column in the dataframe
dataframe.head()[0]

Output:

'1'

Example 3: Extract a single value using head().

Python3




# extract single value based
# on column in the dataframe
dataframe.head()[2]

Output:

'vignan'



My Personal Notes arrow_drop_up
Recommended Articles
Page :