How take a random row from a PySpark DataFrame?

In this article, we are going to learn how to take a random row from a PySpark DataFrame in the Python programming language.

Method 1 : PySpark sample() method

PySpark provides various methods for Sampling which are used to return a sample from the given PySpark DataFrame.

Here are the details of the sample() method : 

Syntax : DataFrame.sample(withReplacement,fractionfloat,seed)

It returns a subset of the DataFrame.

Parameters : 

withReplacement : bool, optional

Sample with replacement or not (default False).

fractionfloat : optional

Fraction of rows to generate

seed : int, optional

Used to reproduce the same random sampling.

Example:

In this example, we need to add a fraction of float data type here from the range [0.0,1.0]. Using the formula : 

Number of rows needed = Fraction * Total Number of rows

We can say that the fraction needed for us is 1/total number of rows.

Python

# importing the library and
# its SparkSession functionality
import pyspark
from pyspark.sql import SparkSession

# creating a session to make DataFrames
random_row_session = SparkSession.builder.appName(
    'Random_Row_Session'
).getOrCreate()

# Pre-set data for our DataFrame
data = [['a', 1], ['b', 2], ['c', 3], ['d', 4]]
columns = ['Letters', 'Position']

# Creating a DataFrame
df = random_row_session.createDataFrame(data,
                                        columns)

# Printing the DataFrame
df.show()

# Taking a sample of df and storing it in #df2
# please not that the second argument here is a fraction
# of the data set we need(fraction is in float)
# number of rows = fraction * total number of rows
df2 = df.sample(False, 1.0/len(df.collect()))

# printing the sample row which is a DataFrame
df2.show()

Output : 

+-------+--------+
|Letters|Position|
+-------+--------+
|      a|       1|
|      b|       2|
|      c|       3|
|      d|       4|
+-------+--------+

+-------+--------+
|Letters|Position|
+-------+--------+
|      b|       2|
+-------+--------+

Method 2: Using takeSample() method

We first convert the PySpark DataFrame to an RDD. Resilient Distributed Dataset (RDD) is the most simple and fundamental data structure in PySpark. They are immutable collections of data of any data type.

We can get RDD of a Data Frame using DataFrame.rdd and then use the takeSample() method. 

Syntax of takeSample() : 

takeSample(withReplacement, num, seed=None) 

Parameters : 

withReplacement : bool, optional

Sample with replacement or not (default False).

num : int

the number of sample values

seed : int, optional

Used to reproduce the same random sampling.

Returns : It returns num number of rows from the DataFrame.

Example: In this example, we are using takeSample() method on the RDD with the parameter num = 1 to get a Row object. num is the number of samples.

Python

# importing the library and
# its SparkSession functionality
import pyspark
from pyspark.sql import SparkSession
from pyspark.sql import Row

# creating a session to make DataFrames
random_row_session = SparkSession.builder.appName(
    'Random_Row_Session'
).getOrCreate()

# Pre-set data for our DataFrame
data = [['a', 1], ['b', 2], ['c', 3], ['d', 4]]
columns = ['Letters', 'Position']

# Creating a DataFrame
df = random_row_session.createDataFrame(data,
                                        columns)

# Printing the DataFrame
df.show()

# Getting RDD object from the DataFrame
rdd = df.rdd

# Taking a single sample of from the RDD
# Putting num = 1 in the takeSample() function
rdd_sample = rdd.takeSample(withReplacement=False,
                            num=1)
print(rdd_sample)

Output : 

+-------+--------+
|Letters|Position|
+-------+--------+
|      a|       1|
|      b|       2|
|      c|       3|
|      d|       4|
+-------+--------+

[Row(Letters='c', Position=3)]

Method 3: Convert the PySpark DataFrame to a Pandas DataFrame and use the sample()  method

We can use toPandas() function to convert a PySpark DataFrame to a Pandas DataFrame. This method should only be used if the resulting Pandas’ DataFrame is expected to be small, as all the data is loaded into the driver’s memory. This is an experimental method.

We will then use the sample() method of the Pandas library. It returns a random sample from an axis of the Pandas DataFrame.

Syntax : PandasDataFrame.sample(n=None, frac=None, replace=False, weights=None, random_state=None, axis=None, ignore_index=False)

Example:

In this example, we will be converting our PySpark DataFrame to a Pandas DataFrame and using the Pandas sample() function on it. 

Python

# importing the library and
# its SparkSession functionality
import pyspark
from pyspark.sql import SparkSession

# creating a session to make DataFrames
random_row_session = SparkSession.builder.appName(
    'Random_Row_Session'
).getOrCreate()

# Pre-set data for our DataFrame
data = [['a', 1], ['b', 2], ['c', 3], ['d', 4]]
columns = ['Letters', 'Position']

# Creating a DataFrame
df = random_row_session.createDataFrame(data,
                                        columns)

# Printing the DataFrame
df.show()

# Converting the DataFrame to
# a Pandas DataFrame and taking a sample row
pandas_random = df.toPandas().sample()

# Converting the sample into
# a PySpark DataFrame
df_random = random_row_session.createDataFrame(pandas_random)

# Showing our randomly selected row
df_random.show()

Output : 

+-------+--------+
|Letters|Position|
+-------+--------+
|      a|       1|
|      b|       2|
|      c|       3|
|      d|       4|
+-------+--------+

+-------+--------+
|Letters|Position|
+-------+--------+
|      b|       2|
+-------+--------+