Memory Efficient Python List Creation in Case of Generators

We have the task of writing memory-efficient Python list creation in the case of Generators and printing the result. In this article, we will demonstrate memory-efficient Python list creation using Generators.

What are Generators?

Generators in Python are a type of iterable, allowing the creation of iterators in a memory-efficient manner. Unlike lists that store all elements in memory at once, generators produce values on the fly as requested. They are defined using a function with the “the ” keyword, which temporarily suspends the function’s state, allowing it to resume from where it left off when the next value is requested.

Memory Efficient Python List Creation in Case of Generators

Below are the examples of Memory Efficient Python demonstrating List Creation in the Case of Generators:

Basic Generator Function

In this example, the below code is used to handle large datasets and demonstrate Python’s generator and list comprehension capabilities. The function generate_numbers(n) yields numbers from 0 to n-1. A list comprehension creates num_list by iterating over this generator. Printing the first 10 elements of num_list is achieved using slicing, selecting elements from index 0 to 9.

Python3

# Define a generator function
def generate_numbers(n):
    for i in range(n):
        yield i


# Create a list using list comprehension
num_list = [num for num in generate_numbers(1000000)]
# Print the first 10 elements of the list
print(num_list[:10])

Output

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Example 1: Memory Efficient Python List Comprehension

Below code imports the sys module and defines a generator function `generate_numbers` that yields numbers from 0 to `n-1`. It then creates a list `num_list_comp` using list comprehension to store the generated numbers, and prints the memory usage of this list using `sys.getsizeof()`.

Python3

import sys

# Define a generator function
def generate_numbers(n):
    for i in range(n):
        yield i

# Memory usage with list comprehension
num_list_comp = [num for num in generate_numbers(1000000)]
print("Memory usage with list comprehension:", sys.getsizeof(num_list_comp))

Output

Memory usage with list comprehension: 8697472

Example 2: Memory Efficient Python List Creation Using Generator

Below code imports the sys module and defines a generator function `generate_numbers` that yields numbers from 0 to `n-1`. It then creates a list `num_list` by converting the generator output directly into a list using the `list()` function, and prints the memory usage of this list using `sys.getsizeof()`.

Python3

import sys

# Define a generator function
def generate_numbers(n):
    for i in range(n):
        yield i


# Memory usage with traditional list creation
num_list = list(generate_numbers(1000000))
print("Memory usage with traditional list creation:", sys.getsizeof(num_list))

Output

Memory usage with traditional list creation: 8250176

Generator vs. List Creation

In this example, below code compares the memory usage of two methods for generating a list of numbers. The first method uses a generator function and a list comprehension, while the second method directly converts the generator to a list. The script prints the memory usage of both approaches and calculates the difference in memory consumption, providing insights into the memory efficiency of each method for generating a list of numbers.

Python3

import sys

# Generator function
def generate_numbers(n):
    for i in range(n):
        yield i

# List comprehension method
num_list_comp = [i for i in generate_numbers(10000)]
size_list_comp = sys.getsizeof(num_list_comp)

# Direct conversion method
num_list = list(generate_numbers(10000)) 
size_list = sys.getsizeof(num_list)

# Calculate the difference
difference = size_list_comp - size_list

# Printing memory usage and difference
print("Memory usage of list comprehension:", size_list_comp, "bytes")
print("Memory usage of Generator:", size_list, "bytes")
print("Memory usage difference:", difference, "bytes")

Output

Memory usage of list comprehension: 87632 bytes
Memory usage of Generator: 83120 bytes
Memory usage difference: 4512 bytes