Parallel Programming with NumPy and SciPy

Parallel computing is a type of computation in which many calculations or the execution of processes are carried out simultaneously. Large problems can often be divided into smaller ones, which can then be solved at the same time.

Required Modules:

pip install scipy
pip install numpy
pip install cupy

Parallel Programming with NumPy 

NumPy is a popular numeric computation library for Python known for its efficient array operations and support for vectorized operations. One way to further optimize NumPy code is to use parallel programming techniques, which take advantage of multiple CPU cores to perform calculations faster.

Parallel dot product calculation using NumPy

First, we have to import the Numpy using Import Numpy as np. Then we have to Create two random vectors a and b of length 100000 and calculate the dot product using NumPy’s built-in parallelization, i.e. np.dot(a,b). Finally, Print the result.

Python3

import numpy as np

# Create two random vectors of length 100000
a = np.random.rand(100000)
b = np.random.rand(100000)

# Calculate the dot product using NumPy's built-in parallelization
dot_product = np.dot(a, b)

print(dot_product)

Output: 25016.0204799

Parallel matrix multiplication using NumPy and Multiprocessing

First, we have to import the Numpy using Import Numpy as np. Then we imported the multiprocessing using the Import pool from multiprocessing. We have defined matrix multiplication as matrix_multiply(args). we have created two random matrices A and B of size 1000×1000 further we Split the matrices into four parts and created a multiprocessing pool with four workers. Then we have to Map the matrix multiplication function to the four parts of the matrices. Concatenate the parts of the result matrix.

Python3

import numpy as np
from multiprocessing import Pool

# Define the matrix multiplication function
def matrix_multiply(args):
    A, B = args
    return np.dot(A, B)

# Create two random matrices of size 1000x1000
A = np.random.rand(1000, 1000)
B = np.random.rand(1000, 1000)

# Split the matrices into 4 parts
A_parts = np.array_split(A, 4, axis=1)
B_parts = np.array_split(B, 4)

# Create a multiprocessing pool with 4 workers
pool = Pool(4)

# Map the matrix multiplication function to the 4 parts of the matrices
C_parts = pool.map(matrix_multiply,
      [(A_part, B_part) for A_part, B_part in zip(A_parts, B_parts)])

# Concatenate the parts of the result matrix
C = np.concatenate(C_parts, axis=1)

print(C)

Output:

 [[ 246.26109895  245.27979434  247.53272716 ...,  246.54602696   246.56427344  247.98649696]
         [ 250.3441429   249.08795621  250.72384067 ...,  250.04416057   250.39319075  251.28326167]
         [ 248.44163838  247.48820248  249.19031327 ...,  248.48692097   249.24465987  250.2703185 ]
         ..., 
         [ 252.35223132  250.92852728  251.9176228  ...,  251.5751485   253.00980032  252.06391074]
         [ 251.8001927   249.67594552  250.62393445 ...,  249.82225854   252.16903134  251.53323254]
         [ 252.24630379  251.09158312  251.64857194 ...,  251.07993262   252.88783961  252.44037699]]

GPU Computing using NumPy

However, NumPy or SciPy alone cannot perform GPU computing. For this, we need some other libraries such as CuPy or PyTorch in addition to NumPy or SciPy to perform GPU computing. At first, we imported the Cupy and Numpy using Import Cupy as cp and Numpy as np. Then we created a random array on GPU, we performed element-wise squaring on the GPU. Then, Transfer the result back to the CPU as a NumPy array. Then we performed further computations on the CPU using NumPy.

Python3

import cupy as cp
import numpy as np

# Create a random array on the GPU
a_gpu = cp.random.rand(3, 3)

# Perform element-wise squaring on the GPU
a_squared_gpu = cp.square(a_gpu)

# Transfer the result back to the CPU as a NumPy array
a_squared_cpu = cp.asnumpy(a_squared_gpu)

# Perform further computations on the CPU using NumPy
a_sum = np.sum(a_squared_cpu)

print("Original array on GPU:")
print(a_gpu)

print("Squared array on GPU:")
print(a_squared_gpu)

print("Squared array on CPU (as NumPy array):")
print(a_squared_cpu)

print("Sum of squared array on CPU (computed using NumPy):")
print(a_sum)

Output:

 Original array on GPU:
        [[0.86840887 0.46334445 0.07575684]
         [0.95068822 0.27356767 0.04985629]
         [0.46676109 0.92671615 0.43278567]]
        Squared array on GPU:
        [[7.53992218e-01 2.14645385e-01 5.71954152e-03]
         [9.04904051e-01 7.48442324e-02 2.48564982e-03]
         [2.17650008e-01 8.58339857e-01 1.87155546e-01]]
        Squared array on CPU (as NumPy array):
        [[7.53992218e-01 2.14645385e-01 5.71954152e-03]
         [9.04904051e-01 7.48442324e-02 2.48564982e-03]
         [2.17650008e-01 8.58339857e-01 1.87155546e-01]]
        Sum of squared array on CPU (computed using NumPy):
        3.628021118080383

Multi-threading using NumPy

We need to import the necessary modules – NumPy and ThreadPoolExecutor from concurrent. futures. Next, we’ll define a function func(x) that we want to execute in parallel. We’ll create an input array arr of values that we want to apply this function to. We’ll also define the number of threads we want to use for parallel execution.

Now, we’ll create a ThreadPoolExecutor with the specified number of threads. This executor will allow us to run the function func on the input array arr in parallel. We’ll use the map method of the executor to apply func to each element of arr in parallel. The map method returns an iterator that contains the results of applying func to each element of arr. To get the actual results, we’ll need to convert the iterator to a NumPy array. We’ll do this by calling the list function on the iterator to get a list of the results and then converting that list to a NumPy array.

Python3

import numpy as np
from concurrent.futures import ThreadPoolExecutor

# Define a function to be executed in parallel
def func(x):
    return x**2

# Create an array of values
arr = np.arange(10)

# Define the number of threads to use
num_threads = 4

# Create a ThreadPoolExecutor with the specified number of threads
with ThreadPoolExecutor(max_workers=num_threads) as executor:
    # Use the executor to map the function to the array in parallel
    results = executor.map(func, arr)

# Convert the results to a NumPy array
results = np.array(list(results))

# Print the input array and the corresponding results
print("Input Array: ", arr)
print("Results: ", results)

Output: 

Finally, we’ll print both the input array arr and the corresponding results array.

Input Array:  [0 1 2 3 4 5 6 7 8 9]
Results:  [ 0  1  4  9 16 25 36 49 64 81]

Parallel Programming with SciPy 

SciPy is a popular Python library for scientific and mathematical calculations. It provides many powerful tools for data analysis and signal processing optimization. In such cases, you can use external libraries and tools to run concurrently in SciPy

Parallelizing a simple map or reducing operation using SciPy’s ‘dask‘ module

At first, we imported the Numpy and Dask using Import Numpy as np and Import dask.array as da. Then we created a random array of size 10000 and chunks of size 1000 and assign it to the x. Then we created an operation and assign it to y.

Print the result.

Python3

import numpy as np
import dask.array as da

x = da.random.normal(size=(10000, 10000), chunks=(1000, 1000))
y = (x + x.T) - x.mean(axis=0)

result = y.sum()

print(result.compute())

Output: -10909.686111782875 

Parallelizing a numerical integration using SciPy’s ‘quad‘ function and the ‘multiprocessing’ module

At first, we imported SciPy and multiprocessing using Import integrate form SciPy and Import multiprocessing. We have created a function using F(x) which calculates the square of the number. Then we created the pool worker and assign it to the result.

Python3

from scipy import integrate
import multiprocessing

def f(x):
    return x**2

pool = multiprocessing.Pool(processes=4)

result = integrate.quad(f, 0, 1)

print(result[0])

Output: 0.33333333333333337