Python doesn’t enable parallelism with for loops by default, but this doesn’t mean it isn’t possible. Read on to find out how to parallel for loop in Python in different ways.

Parallel for Loop In Python

With multiprocessing

The multiprocessing package can spawn a Python process in the same manner as the threading module. It supports both remote and local concurrency.

To sidestep the Global Interpreter Lock of Python, this package doesn’t use threads but opts for subprocesses instead. Thanks to this mechanism, Python developers can fully take advantage of any system’s processors. The package is compatible with Windows and Unix.

Keep in mind that the multiprocessing package does more than just mimic the threading module. For instance, the Pool object is among several new APIs available on this module. It provides a convenient way to parallelize the execution of a function (such as for loops) with several input values. Pool objects also support data parallelism, meaning it can distribute input data across multiple processes as well.

This is an example of how to use the multiprocessing module:

from multiprocessing import Pool
def f(x):
    return x*x
if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(f, [3, 4, 5]))

Output:

[9, 16, 25]

The program creates a Pool object p, which is then run through every element of a list.

This is the equivalent code that uses only a regular for loop:

res = []

for i in [3, 4, 5]:
    res.append(i*i)

print(res)

Output:

[9, 16, 25]

This verifies that both versions return the same results, and the traditional use of for loops even demands a shorter code. However, you won’t take advantage of all processor cores in your system.

With Joblib

Joblib is a third-party Python library providing tools for lightweight pipelining. In particular, it comes with a helper class that can be used to parallelize for loops. The basic idea of this multiprocessing approach is to use generator expressions to convert the code to parallel computing.

Install Joblib if you haven’t (learn how to install packages in Python with this guide):

pip install joblib

This is how you can rewrite the loop above using the Joblib module:

from joblib import Parallel, delayed

def f(x):
    return x*x

Parallel(n_jobs=2)(delayed(f)(i) for i in [3, 4, 5])

Output:

[9, 16, 25]

By giving the n_jobs parameter the value of 2, you specify the number of concurrently running jobs you want the computer to execute at the same time. The delayed() function creates tuples of the function f and all parameters, which will be passed to the interpreter.

The helper class Parallel does the bulk of the job. By default, it creates multiple worker processes to run your code on separate processor cores concurrently.

This default setting works fine most of the time. But it can create huge overhead when the processes need to serialize input and/or output data. In that case, you may need to tell the Parallel() helper to use threads more efficiently by adding the threads value to the prefer parameter like this:

Parallel(n_jobs=2, prefer="threads")(delayed(f)(i) for i in [3, 4, 5])

Joblib uses the ‘loky’ backend module to get the job done by default. But you can switch back to the multiprocessing backend on UNIX systems. This is a process-based backend, and it uses Pool objects from the multiprocessing module under the hood. We don’t recommend this option because it is less robust than the default backend loky.

Parallel(n_jobs=2, backend="multiprocessing")(delayed(f)(i) for i in [3, 4, 5])

It is important to note that computation parallelism makes use of multiple processor cores to carry out several tasks simultaneously. If we execute more processes than the number of cores on the system, each process will see its performance degraded because it benefits from less computational power.

Conclusion

There are several libraries that can help you parallel for loop in Python. They take advantage of the computational power of more CPUs to execute your code faster.