Skip to content
Related Articles

Related Articles

Monitoring memory usage of a running Python program
  • Last Updated : 24 Feb, 2021

Managing memory is important in any programming logic but this becomes necessary for python. As python is used in Ml and AI where vast data are used which needs to be managed. Memory leaks,i.e, the program is out of memory after running for several hours. To manage this memory leaks memory monitoring is essential. Monitoring memory is also called as profiling. As a developer it’s a necessity that we profile our program and use less memory allocation as much as possible.

Method 1: Using Tracemalloc

Tracemalloc is a library module that traces every memory blocks in python. The tracing starts by using the start() during runtime. This library module can also give information about the total size, number and average size of allocated memory blocks. Lets its function with a proper example-

Python3




# importing the module
import tracemalloc
  
# code or function for which memory
# has to be monitored
def app():
    lt = []
    for i in range(0, 100000):
        lt.append(i)
  
# starting the monitoring
tracemalloc.start()
  
# function call
app()
  
# displaying the memory
print(tracemalloc.get_traced_memory())
  
# stopping the library
tracemalloc.stop()

Output:

The output is given in form of (current,peak),i.e, current memory is the memory the code is currently using and peak memory is the maximum space the program used while executing.



(0,3617252)

Method 2: Using Psutil

Psutil is python system library used to keep track of various resources in the system and its utilization. The library is used for  profiling, limiting and management of process resources. In order to install this  do the following-

sudo pip install psutil [Linux]
pip install psutill [Windows]

Let’s see this with an example. All you have to do is add the decorator function and the process_memory function in your code and this will give you the memory consumed by the code and its before and after.

Python3




# importing libraries
import os
import psutil
  
# inner psutil function
def process_memory():
    process = psutil.Process(os.getpid())
    mem_info = process.memory_info()
    return mem_info.rss
  
# decorator function
def profile(func):
    def wrapper(*args, **kwargs):
  
        mem_before = process_memory()
        result = func(*args, **kwargs)
        mem_after = process_memory()
        print("{}:consumed memory: {:,}".format(
            func.__name__,
            mem_before, mem_after, mem_after - mem_before))
  
        return result
    return wrapper
  
# instantiation of decorator function
@profile
  
# main code for which
# memory has to be monitored
def func():
    x = [1] * (10 ** 7)
    y = [2] * (4 * 10 ** 8)
    del x
    return y
  
func()

Output:

func: consumed memory: 307,261,440

Method 3: Using the classic memory profiler 

Memory profiler form pypi is a python library module used for monitoring process memory. It uses psutil code to create decorator and then using it to get the memory distribution. With this pypi module by importing one can save lines and directly call the decorator. To install use the following-

pip install -U memory_profiler

Let’s see this through a code-

Python3




# importing the library
from memory_profiler import profile
  
# instantiating the decorator
@profile
# code for which memory has to
# be monitored
def my_func():
    x = [x for x in range(0, 1000)]
    y = [y*100 for y in range(0, 1500)]
    del x
    return y
  
if __name__ == '__main__':
    my_func()

Output:

The output displays the memory consumed  by each line in the code. Implementation of finding the memory consumption is very easy using memory profiler as we directly call the decorator instead of writing a whole new code. 

Line #    Mem usage    Increment  Occurences   Line Contents
============================================================
     2     30.5 MiB     30.5 MiB           1   @profile
     3                                         def my_func():
     4     30.6 MiB      0.1 MiB        1003       x = [x for x in range(0,1000)]
     5     30.7 MiB      0.1 MiB        1503       y = [y*100 for y in range(0,1500)]
     6     30.7 MiB      0.0 MiB           1       del x
     7     30.7 MiB      0.0 MiB           1       return y  

Attention geek! Strengthen your foundations with the Python Programming Foundation Course and learn the basics.

To begin with, your interview preparations Enhance your Data Structures concepts with the Python DS Course.

My Personal Notes arrow_drop_up
Recommended Articles
Page :