Skip to content
Related Articles

Related Articles

Save Article
Improve Article
Save Article
Like Article

How to Check if PySpark DataFrame is empty?

  • Last Updated : 30 May, 2021

In this article, we are going to check if the Pyspark DataFrame or Dataset is Empty or Not.

At first, let’s create a dataframe

 Attention geek! Strengthen your foundations with the Python Programming Foundation Course and learn the basics.  

To begin with, your interview preparations Enhance your Data Structures concepts with the Python DS Course. And to begin with your Machine Learning Journey, join the Machine Learning - Basic Level Course

Python3






# import modules
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType
  
# defining schema
schema = StructType([
    StructField('COUNTRY', StringType(), True),
    StructField('CITY', StringType(), True),
    StructField('CAPITAL', StringType(), True)
])
  
# Create Spark Object
spark = SparkSession.builder.appName("TestApp").getOrCreate()
  
# Create Empty DataFrame with Schema.
df = spark.createDataFrame([], schema)
  
# Show schema and data
df.printSchema()
df.show(truncate=False)

Output:

Checking dataframe is empty or not

We have Multiple Ways by which we can Check :

Method 1: isEmpty()

The isEmpty function of the DataFrame or Dataset returns true when the DataFrame is empty and false when it’s not empty. If the dataframe is empty, invoking “isEmpty” might result in NullPointerException.

Note : calling df.head() and df.first() on empty DataFrame returns java.util.NoSuchElementException: next on empty iterator exception.

Python3




print(df.head(1).isEmpty)
print(df.first(1).isEmpty)
print(df.rdd.isEmpty())

Output:

True
True
True

Method 2: count()

  It calculates the count from all partitions from all nodes

Code:

Python3




print(df.count() > 0)
print(df.count() == 0)
False
True



My Personal Notes arrow_drop_up
Recommended Articles
Page :