Add a column with the literal value in PySpark DataFrame
In this article, we are going to see how to add a column with the literal value in PySpark Dataframe.
Creating dataframe for demonstration:
Python3
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName( "lit_value" ).getOrCreate()
data = spark.createDataFrame([( 'x' , 5 ),( 'Y' , 3 ),
( 'Z' , 5 ) ],[ 'A' , 'B' ])
data.printSchema()
data.show()
|
Output:
Method 1: Using Lit() function
Here we can add the constant column ‘literal_values_1’ with value 1 by Using the select method. The lit() function will insert constant values to all the rows.
Select table by using select() method and pass the arguments first one is the column name, or “*” for selecting the whole table and second argument pass the lit() function with constant values.
Python3
from pyspark.sql.functions import lit
df2 = data.select( '*' ,lit( "1" ).alias( "literal_values_1" ))
df2.printSchema()
df2.show()
|
Output:
Method 2: Using SQL clause
In this method first, we have to create the temp view of the table with the help of createTempView we can create the temporary view. The life of this temp is up to the life of the sparkSession. CreateOrReplace will create the temp table if it is not available or if it is available then replace it.
Then after creating the table select the table by SQL clause which will take all the values as a string
Python3
df2.createOrReplaceTempView( "temp" )
df2 = spark.sql( "select *, 2 as literal_values_2 from temp" )
df2.printSchema()
df2.show()
|
Output:
Method 3: Using UDF(User-defined Functions) Method
This function allows us to create the new function as per our requirements that’s why this is also called a user-defined function. Now we define the datatype of the UDF function and create the functions which will return the values in the form of a new column
Python3
from pyspark.sql.functions import udf
@udf ( "int" )
def lit_col():
return 3
df2 = df2.withColumn( 'literal_values_3' , lit_col())
df2.printSchema()
df2.show()
|
Output:
Last Updated :
24 Sep, 2021
Like Article
Save Article
Share your thoughts in the comments
Please Login to comment...