Monitoring memory usage of a running Python program
Managing memory is important in any programming logic but this becomes necessary for python. As python is used in Ml and AI where vast data are used which needs to be managed. Memory leaks,i.e, the program is out of memory after running for several hours. To manage these memory leaks memory monitoring is essential. Monitoring memory is also called profiling. As a developer, it’s a necessity that we profile our program and use less memory allocation as much as possible.
Method 1: Using Tracemalloc
Tracemalloc is a library module that traces every memory block in python. The tracing starts by using the start() during runtime. This library module can also give information about the total size, number, and average size of allocated memory blocks. Lets its function with a proper example-
The output is given in form of (current, peak),i.e, current memory is the memory the code is currently using and peak memory is the maximum space the program used while executing.
Method 2: Using Psutil
Psutil is a python system library used to keep track of various resources in the system and their utilization. The library is used for profiling, limiting, and management of process resources. In order to install this do the following-
sudo pip install psutil [Linux] pip install psutill [Windows]
Let’s see this with an example. All you have to do is add the decorator function and the process_memory function in your code and this will give you the memory consumed by the code and it’s before and after.
func: consumed memory: 307,261,440
Method 3: Using the classic memory profiler
Memory profiler from PyPI is a python library module used for monitoring process memory. It uses psutil code to create a decorator and then uses it to get the memory distribution. With this pypi module by importing one can save lines and directly call the decorator. To install use the following-
pip install -U memory_profiler
Let’s see this through a code-
The output displays the memory consumed by each line in the code. Implementation of finding the memory consumption is very easy using a memory profiler as we directly call the decorator instead of writing a whole new code.
Line # Mem usage Increment Occurences Line Contents ============================================================ 2 30.5 MiB 30.5 MiB 1 @profile 3 def my_func(): 4 30.6 MiB 0.1 MiB 1003 x = [x for x in range(0,1000)] 5 30.7 MiB 0.1 MiB 1503 y = [y*100 for y in range(0,1500)] 6 30.7 MiB 0.0 MiB 1 del x 7 30.7 MiB 0.0 MiB 1 return y