Skip to content
Related Articles

Related Articles

Improve Article
Save Article
Like Article

Stratified Sampling in Pandas

  • Difficulty Level : Basic
  • Last Updated : 02 Nov, 2021

Stratified Sampling is a sampling technique used to obtain samples that best represent the population. It reduces bias in selecting samples by dividing the population into homogeneous subgroups called strata, and randomly sampling data from each stratum(singular form of strata). 

In statistics, stratified sampling is used when the mean values of each stratum will differ. In Machine learning, stratified sampling is commonly used to create test datasets to evaluate models especially when the dataset is significantly large and unbalanced.  

 Attention geek! Strengthen your foundations with the Python Programming Foundation Course and learn the basics.  

To begin with, your interview preparations Enhance your Data Structures concepts with the Python DS Course. And to begin with your Machine Learning Journey, join the Machine Learning - Basic Level Course



Steps involved in stratified sampling

  • Separating the Population into Strata: In this step, the population is divided into strata based on similar characteristics and every member of the population must belong to exactly one stratum (singular of strata).
  • Determine the sample size: Decide how small or large the sample should be.
  • Randomly sampling each stratum: Random samples from each stratum are selected using either Disproportionate sampling where the sample size of each stratum is equal irrespective of the population size of the stratum or Proportionate sampling where the sample size of each stratum is proportional to the population size of the stratum.

Example 1: 

In this example, we have a dummy dataset of 10 students and we will sample out 6 students based on their grades, using both disproportionate and proportionate stratified sampling.

Step 1: Create the dummy dataset from a python dictionary using pandas DataFrame 

Python3




import pandas as pd
  
# Create a dictionary of students
students = {
    'Name': ['Lisa', 'Kate', 'Ben', 'Kim', 'Josh',
             'Alex', 'Evan', 'Greg', 'Sam', 'Ella'],
    'ID': ['001', '002', '003', '004', '005', '006'
           '007', '008', '009', '010'],
    'Grade': ['A', 'A', 'C', 'B', 'B', 'B', 'C'
              'A', 'A', 'A'],
    
    'Category': [2, 3, 1, 3, 2, 3, 3, 1, 2, 1]
}
  
# Create dataframe from students dictionary
df = pd.DataFrame(students)
  
# view the dataframe
df

Output:

Notice that there are 50% grade A students, 30% grade B students, and 20% grade C students.

Step 2: Create a sample of 6 students disproportionately (equal number of students from each grade stratum)

Disproportionate Sampling: Using pandas groupby, separate the students into groups based on their grade i.e A, B, C and randomly sample 2 students from each grade group using the sample function



Python3




df.groupby('Grade', group_keys=False).apply(lambda x: x.sample(2))

Output:

Step 3: Sample out 60% of students proportionately (create proportional samples from each stratum based on its proportion in the population)

Proportionate Sampling: Using pandas groupby, separate the students into groups based on their grade i.e A, B, C, and random sample from each group based on population proportion. The total sample size is 60%(0.6) of the population

Python3




df.groupby('Grade', group_keys=False).apply(lambda x: x.sample(frac=0.6))

Output: 

Notice that even in the sample, there are 50% grade A students, 30% grade B students, and 20% grade C students.

Example 2: 

In this example, we will be creating sample data from the train dataset. Titanic was a British passenger that sank in the North Atlantic Ocean after striking an iceberg. The dataset contains information on all passengers who boarded the Titanic, a passenger either died or survived the crash, so we will be using the Survived column as our stratifying column.



Step 1: Read in the dataset from the CSV file

Python3




import pandas as pd
  
# read the dataset as csv file
data = pd.read_csv('Titanic.csv')
  
# drop the name column as it is of no importance here
data.drop('Name', axis=1, inplace=True)
  
# view the first 5 rows of the titanic dataset
data.head()

Output:

Step 2: Check the percentage of dead/survived passengers

Check the proportion/percentage of passengers who died or survived this is given a number of dead or alive passengers / total number of passengers * 100

Python3




(data['Survived'].value_counts()) / len(data) * 100

Output:

0    61.616162
1    38.383838

where 0 represents passengers that died (61.6%) and 1 represents passengers that survived (38.4%)

Step 3: Disproportionately sample out  8 passengers (4 who died and 4 who survived)



Python3




# Disproportionate sampling:
# randomly select 4 samples from each stratum
  
data.groupby('Survived', group_keys=False).apply(lambda x: x.sample(4))

Output:

Step 4: Proportionately sample out  1%(0.01)  of passengers (0.6% died and 0.4% survived)

Python3




data.groupby('Survived', group_keys=False).
apply(lambda x: x.sample(frac=0.01))

Output:




My Personal Notes arrow_drop_up
Recommended Articles
Page :

Start Your Coding Journey Now!