How to Write Memory Efficient Loops in Python

When you are working on projects with limited resources or dealing with large amounts of data, it is important to be mindful of how much memory your Python code is using. In Python, loops are fundamental parts of programs, and making them use less memory can make your code run faster. This article explains how to make loops that use less memory helping you with large amounts of data more effectively.

How to Write Memory Efficient Loops in Python?

Below, are the code examples of how to write memory-efficient loops in Python.

• Regular Loop (Memory-Inefficient)
• Generator Approach (Memory-Efficient)
• List Comprehension (Memory-Efficient)

Regular Loop (Memory-Inefficient)

In this example, the below code defines a function regular_loop that iterates over a given list, performing an operation (multiplying each item by 2) and storing the result in a new list. It then measures the memory usage before and after applying the function to a large sample data (a list of 1 million integers), demonstrating the memory impact of the operation.

Python3

import sys

def regular_loop(data):
    """Regular loop that creates a new list within the loop."""
    result = []
    for item in data:
        result.append(item * 2)  # Example operation
    return result

# Sample data (adjust size for memory impact)
data = list(range(1000000))  # 1 million integers

# Measure memory usage before and after
memory_before = sys.getsizeof(data)
result = regular_loop(data)
memory_after = sys.getsizeof(result)

print("Regular Loop Memory Usage:")
print(f"Before: {memory_before} bytes")
print(f"After: {memory_after} bytes")
print(f"Difference: {memory_after - memory_before} bytes\n")

Output

Regular Loop Memory Usage:
Before: 9000120 bytes
After: 8697472 bytes
Difference: -302648 bytes

Generator Approach (Memory-Efficient)

In this example, below code defines a generator function generator_approach that yields values one at a time, avoiding the creation of a separate data structure. It measures memory usage before and after using the generator to process a sample data set (a list of integers), demonstrating minimal memory impact compared to traditional methods.

Python3

import sys

def generator_approach(data):
    """Generator function that yields values one at a time."""
    for item in data:
        yield item * 2  # Example operation

data = [1, 2, 3, 4, 5]  # Sample data

# Measure memory usage (generators don't create separate data structures)
memory_before = sys.getsizeof(data)
result = generator_approach(data)  # `result` is a generator object

# Accessing generator values (simulates usage)
for item in result:
    pass  # Example processing

memory_after = sys.getsizeof(data)  # Get size of original data

print("Generator Approach Memory Usage:")
print(f"Before: {memory_before} bytes")
print(f"After: {memory_after} bytes")  # Same as original data size
print(f"Difference: {memory_after - memory_before} bytes\n")

Output

Generator Approach Memory Usage:
Before: 112 bytes
After: 112 bytes
Difference: 0 bytes

List Comprehension (Memory-Efficient)

In this example, below code defines a function `list_comprehension` that creates a new list by doubling each item in the input list using list comprehension. It measures memory usage before and after applying list comprehension to a sample data set, showcasing the memory impact of the operation.

Python3

import sys

def list_comprehension(data):
    """List comprehension creates a new list concisely."""
    return [item * 2 for item in data]

data = [1, 2, 3, 4, 5]  # Sample data

# Measure memory usage
memory_before = sys.getsizeof(data)
result = list_comprehension(data)
memory_after = sys.getsizeof(result)

print("List Comprehension Memory Usage:")
print(f"Before: {memory_before} bytes")
print(f"After: {memory_after} bytes")
print(f"Difference: {memory_after - memory_before} bytes\n")

Output

List Comprehension Memory Usage:
Before: 112 bytes
After: 136 bytes
Difference: 24 bytes