Drop a column with same name using column index in PySpark

In this article, we are going to learn how to drop a column with the same name using column index using Pyspark in Python.

Pyspark offers you the essential function ‘drop‘ through which you can easily delete one or more columns. But have you ever got the requirement in which you have various columns with the same column names and the requirement is to delete all the duplicate columns? This can be achieved in Pyspark by obtaining the column index of all the columns with the same name and then deleting those columns using the drop function.

Example 1:

In the example, we have created a data frame with four columns ‘name‘, ‘marks‘, ‘marks‘, ‘marks‘ as follows:

Once created, we got the index of all the columns with the same name, i.e., 2, 3, and added the suffix ‘_duplicate‘ to them using a for a loop. Finally, we removed the columns with suffixes ‘_duplicate‘ in them and displayed the data frame.

Python3

# Python program to drop a column with same
# name using column index in PySpark

# Import the library SparkSession
from pyspark.sql import SparkSession

# Create a spark session using getOrCreate() function
spark_session = SparkSession.builder.getOrCreate()

# Create a data frame with duplicate column names
df = spark_session.createDataFrame(
  [('Arun',1,2,3),('Aniket',4,5,6),
                  ('Ishita',7,8,9)],
  ['name','marks','marks','marks'])

# Store all the column names in the list
df_cols = df.columns

# Get index of the duplicate columns
duplicate_col_index = [idx for idx,
  val in enumerate(df_cols) if val in df_cols[:idx]]

# Create a new list by renaming duplicate
# columns by adding prefix '_duplicate'
for i in duplicate_col_index:
    df_cols[i] = df_cols[i] + '_duplicate'

# Rename the duplicate columns in data frame
df = df.toDF(*df_cols)

# Create a list for the columns to be removed
cols_to_remove =

# Remove the columns with same name
df.drop(*cols_to_remove).show()

Output:

Example 2:

In the example, we have created a data frame with five columns with names ‘day’, ‘temperature‘, ‘temperature‘, ‘temperature‘, and ‘temperature‘ as follows:

Once created, we got the index of all the columns with the same name, i.e., 2, 3, 4, and added the prefix ‘day_‘ to them using a for loop. Finally, we removed the columns with the prefixes ‘day_‘ in them and displayed the data frame.

Python3

# Python program to drop a column with same
# name using column index in PySpark

# Import the library SparkSession
from pyspark.sql import SparkSession

# Create a spark session using getOrCreate() function
spark_session = SparkSession.builder.getOrCreate()

# Create a data frame with duplicate column names
df = spark_session.createDataFrame(
  [('Monday',25,27,29,30),('Tuesday',40,38,36,34),
   ('Wednesday',18,20,22,17),('Thursday',25,27,29,19)],
  ['day','temperature','temperature','temperature',
                                      'temperature'])

# Store all the column names in the list
df_cols = df.columns

# Get index of the duplicate columns
duplicate_col_index = [idx for idx,
   val in enumerate(df_cols) if val in df_cols[:idx]]

# Create a new list by renaming duplicate
# columns by adding prefix 'day_'
for i in duplicate_col_index:
    df_cols[i] = 'day_'+ df_cols[i]

# Rename the duplicate columns in data frame
df = df.toDF(*df_cols)

# Create a list for the columns to be removed
cols_to_remove =

# Remove the columns with same name
df.drop(*cols_to_remove).show()

Output: