Open In App
Related Articles

How to change dataframe column names in PySpark ?

Improve Article
Save Article
Like Article

In this article, we are going to see how to change the column names in the pyspark data frame. 

Let’s create a Dataframe for demonstration:


# Importing necessary libraries
from pyspark.sql import SparkSession
# Create a spark session
spark = SparkSession.builder.appName('pyspark - example join').getOrCreate()
# Create data in dataframe
data = [(('Ram'), '1991-04-01', 'M', 3000),
        (('Mike'), '2000-05-19', 'M', 4000),
        (('Rohini'), '1978-09-05', 'M', 4000),
        (('Maria'), '1967-12-01', 'F', 4000),
        (('Jenis'), '1980-02-17', 'F', 1200)]
# Column names in dataframe
columns = ["Name", "DOB", "Gender", "salary"]
# Create the spark dataframe
df = spark.createDataFrame(data=data,
# Print the dataframe

Output :

Method 1: Using withColumnRenamed()

We will use of withColumnRenamed() method to change the column names of pyspark data frame.

Syntax: DataFrame.withColumnRenamed(existing, new)


  • existingstr: Existing column name of data frame to rename.
  • newstr: New column name.
  • Returns type: Returns a data frame by renaming an existing column.

Example 1: Renaming the single column in the data frame

Here we’re Renaming the column name ‘DOB’ to ‘DateOfBirth’.


# Rename the column name from DOB to DateOfBirth
# Print the dataframe

Output :

Example 2: Renaming multiple column names


# Rename the column name 'Gender' to 'Sex'
# Then for the returning dataframe
# again rename the 'salary' to 'Amount'

Output :

Method 2: Using selectExpr()

Renaming the column names using selectExpr() method

Syntax : DataFrame.selectExpr(expr)

Parameters :

expr : It’s an SQL expression.

Here we are renaming Name as a name.


# Select the 'Name' as 'name'
# Select remaining with their original name
data = df.selectExpr("Name as name","DOB","Gender","salary")
# Print the dataframe

Output :

Method 3: Using select() method


Parameters :

cols: List of column names as strings.

Return type: Selects the cols in the dataframe and returns a new DataFrame.

Here we Rename the column name ‘salary’ to ‘Amount’


# Import col method from pyspark.sql.functions
from pyspark.sql.functions import col
# Select the 'salary' as 'Amount' using aliasing
# Select remaining with their original name
data ="Name"),col("DOB"),
# Print the dataframe

Output :

Method 4: Using toDF()

This function returns a new DataFrame that with new specified column names.

Syntax: toDF(*col)

Where, col is a new column name

In this example, we will create an order list of new column names and pass it into toDF function


Data_list = ["Emp Name","Date of Birth",
             " Gender-m/f","Paid salary"]
new_df = df.toDF(*Data_list)


Whether you're preparing for your first job interview or aiming to upskill in this ever-evolving tech landscape, GeeksforGeeks Courses are your key to success. We provide top-quality content at affordable prices, all geared towards accelerating your growth in a time-bound manner. Join the millions we've already empowered, and we're here to do the same for you. Don't miss out - check it out now!

Last Updated : 15 Feb, 2022
Like Article
Save Article
Similar Reads
Complete Tutorials