Abstract flowing gradient in deep indigo and blue tones, smooth and luminous, evoking a modern digital learning atmosphere

Computer Science and programming articles. We do not sell courses.

How to use Python's multiprocessing for parallel computing

Python remains the dominant language for scripting in Australian university CS programs and in research labs from Sydney to Perth, yet anyone who has tried to speed up a CPU-heavy task with threads has hit the same wall. The Global Interpreter Lock allows only one thread to execute Python bytecode at a time, so pure-Python threads cannot exploit extra cores. The standard library's multiprocessing module solves this by launching separate Python interpreters, each with its own memory space and its own GIL, so work really does run in parallel.

This article walks through how to use Python's multiprocessing for parallel computing, starting with the basics of spawning workers, moving through the differences between Pool, Queue and Manager, and ending with practical advice for workloads you might actually see in Australia — processing ABS census files, running Monte Carlo simulations for a Sydney fintech, or crunching mining data out of Kalgoorlie. Along the way, you will find a comparison table, two bulleted summaries, and pointers to deeper reading on parallel patterns.

Why the GIL pushes you toward multiprocessing

Python's reference implementation, CPython, serialises access to interpreter state with the Global Interpreter Lock. Even on a 16-core AWS EC2 instance in the ap-southeast-2 Sydney region, threads doing tight numeric loops will not run faster than a single core. I/O-bound work — downloading data from the Bureau of Meteorology, scraping ASIC business names, streaming from Kafka — still benefits from threading or asyncio because the lock is released while waiting. CPU-bound work, such as fitting a gradient boosting model or resampling a raster, needs real parallelism.

multiprocessing was designed precisely for the second case. Each child process receives its own interpreter, so the GIL is no longer shared. The trade-off is that data passed between processes must be serialised with pickle by default, which is slower than sharing a Python object between threads. Engineers at CSIRO's Data61 in Eveleigh routinely combine NumPy arrays with multiprocessing to parallelise satellite-image processing pipelines, because the overhead of copying large buffers is acceptable when the per-image CPU work dominates.

Setting up your first multiprocessing script

The simplest entry point is the Process class. You define a function that takes a single argument — typically a chunk of work or a parameter tuple — then instantiate Process with that function and call .start(). Workers run concurrently and you join them with .join() before reading results.

A working skeleton looks like this in concept:

import multiprocessing as mp

def worker(payload):
    pass

if __name__ == "__main__":
    tasks = [...]
    procs = [mp.Process(target=worker, args=(t,)) for t in tasks]
    for p in procs:
        p.start()
    for p in procs:
        p.join()

The if __name__ == "__main__": guard is not stylistic noise. On Linux — which is what most machines running on NBN-linked home setups in Adelaide or Brisbane will use — and on macOS, child processes re-import the main module. Without the guard, recursive imports cascade and you get the dreaded "freeze_support" or duplicate-process errors. On Windows the guard is enforced by the multiprocessing bootstrap itself.

To capture return values, wrap workers in a Pool, queue results into a multiprocessing.Queue, or return them via a shared Value or Array. The next section compares these patterns side by side.

Comparing Pool, Process, and Manager strategies

Choosing the right primitive saves both code and CPU. Process is a thin wrapper around one worker function call and suits fire-and-forget tasks such as logging rotated files or running independent simulations. Pool pre-spawns a configurable number of workers and exposes map, imap and apply_async, which is convenient for embarrassingly parallel loops. Manager objects (multiprocessing.Manager()) let multiple processes mutate a shared dictionary or list through a proxy server process, at the price of extra latency per access.

Primitive Best for Data sharing Startup cost Typical Australian use case
Process Independent long jobs Queue or return value Simplicity (one worker per call) Running ten independent runs on a UNSW lab workstation
Pool Embarrassingly parallel loops Iterable input, list output Pool warm-up + worker count Batch-processing 200 GB of Bureau of Statistics CSV exports in Melbourne
Manager().dict / list Shared mutable state Proxy over socket Manager proxy process Coordinating a leaderboard across Sydney and Brisbane hackathon teams
SharedMemory Large NumPy arrays Raw shared buffer Initial copy Passing satellite rasters between worker processes at Geoscience Australia

Pool deserves its own discussion because it is the workhorse of batch processing. The class manages the lifecycle of worker processes for you, reuses them across tasks, and gives you several submission styles to choose from.

Common Pool entry points worth memorising:

Pick the primitive based on how the result is consumed. Streaming pipelines favour imap and Queue; bulk batch jobs favour map and Pool; coordinated dashboards favour Manager proxies.

Sharing data between processes safely

Because each process owns its memory, sharing requires explicit mechanics. Three approaches dominate. First, multiprocessing.Value and multiprocessing.Array wrap ctypes primitives and arrays in shared memory; access is fast but you typically need a Lock to prevent races. Second, Manager().dict(), Manager().list() and Manager().Namespace() expose proxy objects so multiple processes can mutate a container, paying for the convenience with cross-process marshalling. Third, since Python 3.8 the SharedMemory class lets you allocate a named block and wrap a NumPy ndarray over the top, which is by far the cheapest way to share multi-megabyte arrays.

A common Australian scenario is a small fintech in the Sydney CBD running a credit-scoring batch overnight. The team loads a 4 GB customer-features table, distributes chunks across 32 workers on an m6i.8xlarge, then collects per-chunk partial sums. Using SharedMemory keeps the 4 GB resident once instead of pickling it 32 times. For broader context beyond parallel computing, browse the curated algorithms index, which covers lock-free queues, concurrent hash tables, and producer-consumer patterns.

Race conditions still happen even with the right primitive. If two workers call a function that reads-modifies-writes a shared counter, you need an explicit multiprocessing.Lock. As a rule of thumb: if you can phrase the work as map and gather results, do that; if you really need shared mutable state, wrap every mutation site with a lock and document it.

Practical performance tips on Australian workloads

Once the basics work, the next question is how to get the most out of the hardware you actually have. Whether you are running on a MacBook Pro in Surry Hills, a beefy tower in a Perth mining-data office, or a cloud VM in the AWS Sydney region, the same principles apply.

Speed-ups you can claim without changing your algorithm:

Two Australian-specific notes are worth adding. Latency between Sydney and the us-east-1 AWS region is around 200 ms, so co-locating your data in ap-southeast-2 saves measurable round-trip time when workers stream from S3. Many research institutions — the University of Melbourne, ANU in Canberra, the Pawsey Supercomputing Centre in Perth — run SLURM-managed clusters where per-task interfaces expect one process per task, and multiprocessing.Pool fits cleanly into the per-node CPU budget. For a broader view of how parallel primitives fit into larger programs, the hello ML homepage links to tutorials that cover both the language fundamentals and the algorithmic patterns underneath them.

Common pitfalls and debugging strategies

Multiprocessing errors are stranger than threading errors because the tracebacks can come from a different interpreter that has already exited. A few patterns come up again and again.

Pickling failures are the most frequent. If you pass a lambda, a local function, a database connection, or a file handle, pickle cannot serialise it and you get a cryptic AttributeError. The fix is to keep the worker function at module scope, pass only primitive types, and reopen any resource inside the worker. Deadlocks happen when every worker is waiting on a Queue that nothing is filling, often because the feeder process crashed silently — always log from inside the worker and capture stderr. Resource leaks occur when a Pool is created but not closed; use Pool as a context manager or call pool.close() followed by pool.join().

For deeper reading on how scheduling data structures shape these patterns, the walkthrough on round-robin scheduling in C shows the underlying data structure that operating-system schedulers use to share CPU time among many workers — the same conceptual problem multiprocessing solves at the process level rather than the thread level. Comparing the two perspectives often clarifies why certain trade-offs exist in Python's design.

Finally, remember that multiprocessing is not always the answer. For I/O-heavy Australian workloads such as pulling weather feeds from the Bureau of Meteorology or scraping ASIC's business registry, asyncio or aiohttp will outperform processes because the bottleneck is the network, not the CPU. Reach for multiprocessing when the work is CPU-bound, the data fits in RAM, and the speedup you want is roughly proportional to the number of cores you have to throw at it.