PySpark – Extracting single value from DataFrame

In this article, we are going to extract a single value from the pyspark dataframe columns. To do this we will use the first() and head() functions.

Single value means only one value, we can extract this value based on the column name

Syntax: 

• dataframe.first()[‘column name’]
• Dataframe.head()[‘Index’]

Where,

• dataframe is the input dataframe and column name is the specific column
• Index is the row and columns.

So we are going to create the dataframe using the nested list.

Python3

# importing module
import pyspark

# importing sparksession from pyspark.sql module
from pyspark.sql import SparkSession

# creating sparksession and giving an app name
spark = SparkSession.builder.appName('sparkdf').getOrCreate()

# list  of students  data
data =[["1","sravan","vignan"],
       ["2","ojaswi","vvit"],
       ["3","rohith","vvit"],
       ["4","sridevi","vignan"],
       ["1","sravan","vignan"],
       ["5","gnanesh","iit"]]

# specify column names
columns=['student ID','student NAME','college']

# creating a dataframe from the lists of data
dataframe = spark.createDataFrame(data,columns)

print("Actual data in dataframe")
# show dataframe
dataframe.show()

Output:

Actual data in dataframe
+----------+------------+-------+
|student ID|student NAME|college|
+----------+------------+-------+
|         1|      sravan| vignan|
|         2|      ojaswi|   vvit|
|         3|      rohith|   vvit|
|         4|     sridevi| vignan|
|         1|      sravan| vignan|
|         5|     gnanesh|    iit|
+----------+------------+-------+

Example 1: Python program to extract a single value from a particular column using first().

Python3

# extract single value based on
# column in the dataframe
dataframe.first()['student ID']

Output:

'1'

Example 2: Extract a single value using head().

Python3

# extract single value based
# on column in the dataframe
dataframe.head()[0]

Output:

'1'

Example 3: Extract a single value using head().

Python3

# extract single value based
# on column in the dataframe
dataframe.head()[2]

Output:

'vignan'