Using Python's multiprocessing module for CPU-bound parallelism
Python developers in Sydney's fintech sector and Brisbane's growing AI startups often hit the same wall: a script that runs perfectly on a 1,000-row sample becomes glacial once the production dataset lands. The culprit is usually the Global Interpreter Lock, which forces Python bytecode through a single thread even on machines with sixteen cores. The multiprocessing module sidesteps this by spawning separate Python interpreters, each with its own memory space and its own GIL.
The trade-off is communication overhead. Threads share memory cheaply; processes do not. For embarrassingly parallel jobs such as grid searches, hyperparameter sweeps, or training the base learners in a bagged ensemble, that overhead is a small price to pay for genuine CPU parallelism. Understanding when multiprocessing fits, and when a thread pool or asyncio loop is the better tool, separates scripts that finish overnight from those that finish in minutes.
Why multiprocessing is the right tool for CPU work
The GIL is the headline reason, but the real story is workload shape. A CPU-bound task spends most of its time computing: matrix multiplications, image transforms, cryptographic hashing, simulation steps. A parallel version can be split into independent chunks that each run on a separate core, and multiprocessing is built exactly for this.
Australian data teams see the effect clearly when crunching Bureau of Meteorology climate grids. A 30-year reanalysis at 5 km resolution over the continent runs to hundreds of millions of cells; processing it on a single core can take days. Splitting the grid by latitude band across twelve worker processes on a multi-core workstation cuts the runtime almost linearly, limited mainly by disk I/O.
Contrast this with I/O-bound work: hitting a third-party API, reading from a slow disk, waiting on a database. There the bottleneck is waiting, not computing, and threads or asyncio will do the job with less memory and zero pickling cost. multiprocessing will still work, but you will pay the serialisation tax for no real speedup.
Building your first parallel pipeline with Pool
The Pool class is the most common entry point. It manages a fixed number of worker processes and exposes a small, sharp API. The classic map call applies a function to every item in an iterable and distributes the work automatically across the pool.
from multiprocessing import Pool
def square(x):
return x * x
if __name__ == "__main__":
with Pool(processes=8) as pool:
results = pool.map(square, range(10_000_000))
For longer-running tasks where you want results as they finish, imap_unordered streams outputs back the moment each worker is done. apply_async submits a single job and returns a handle you can poll later. The asynchronous variants matter when jobs have uneven duration, which is typical in machine-learning workloads where one fold of cross-validation takes far longer than another. Dynamic-programming problems such as how to solve the longest common subsequence with dynamic programming for many input pairs also fit this shape: each worker evaluates a different (s1, s2) tuple on its own core and returns the table dimensions as soon as it finishes.
Process count should usually match the number of physical CPU cores. On a Threadripper workstation in Perth's mining analytics shops, you can inspect the topology with os.cpu_count() and the psutil library, though os.cpu_count() returns logical cores and you may want to halve it on hyper-threaded chips to avoid context-switching noise.
How multiprocessing compares to threading and asyncio
Choosing between the three core concurrency tools is a recurring decision in any Python service. The table below summarises the trade-offs an Australian engineering team might weigh when picking an approach for a new pipeline.
| Aspect | multiprocessing | threading | asyncio |
|---|---|---|---|
| GIL bypass | Yes, separate interpreters | No | No |
| Best for | CPU-bound work | I/O-bound work | High-concurrency I/O |
| Memory overhead | High, each process has its own copy | Low, shared heap | Very low, single thread |
| Communication cost | Pickle serialisation, slow | Shared variables, fast | Queues and events, fast |
| Startup cost | Heavyweight fork or spawn | Lightweight | Lightweight |
| Debugging complexity | Hard, no shared state inspection | Medium, race conditions | Medium, traceable coroutines |
| Cross-platform quirks | spawn default on macOS | Works everywhere | Works everywhere |
The takeaway: reach for multiprocessing when you can split the work into independent chunks and each chunk chews CPU. Reach for threading or asyncio when the bottleneck is waiting. Reach for Dask or Ray when you want to spread the work across multiple machines, which is what teams at Atlassian in Sydney do when a single-node pool runs out of steam.
Sharing data across process boundaries
Workers cannot see each other's variables. Every argument to a worker function is pickled and copied across the process boundary, and every return value is pickled back. For small arguments such as a list of file paths or a config dictionary, this is fine. For a 4 GB numpy array it is a disaster.
Three patterns cover most cases. The first is to keep data on disk and pass only paths, letting each worker read what it needs. The second is to use shared memory via multiprocessing.shared_memory, which lets workers map the same buffer without copying. The third is a Manager, which exposes a proxy object that lives in a server process; workers access it through a pipe. Queues and Pipes handle producer-consumer patterns well: a classic example is a four-worker pool ingesting images from a folder while a separate process writes resized versions to a destination bucket, the kind of pipeline used for image-processing jobs that scale across a fleet of EC2 instances in the AWS Sydney region.
A subtle point worth remembering: on macOS and Windows the default start method since Python 3.4 is spawn, not fork. The worker processes are launched fresh, so anything at module level in your script runs again inside each child. Guard executable code with if __name__ == "__main__":, otherwise you will get infinite recursion the first time you run your script on a colleague's laptop.
Patterns, pitfalls, and a worked example
A few habits make multiprocessing scripts reliable. Profile first with cProfile or py-spy to confirm the workload is genuinely CPU-bound, otherwise the parallelism overhead will slow you down. Reuse pools instead of recreating them in a loop, since worker startup costs dwarf the per-task work for small jobs. Catch exceptions inside the worker function and return them as results, because a traceback printed in a child process is easy to miss.
Embarrassingly parallel problems are everywhere in machine-learning workflows. Training the base learners in bagging and boosting techniques is a textbook case: each tree or boosting round is independent and the training loop parallelises cleanly across cores. A bagging ensemble of decision trees trains each estimator on a bootstrap sample, and the wall-clock improvement from a ProcessPoolExecutor is close to linear up to the number of physical cores.
When the problem no longer splits cleanly, switch libraries. joblib provides a friendlier Pool wrapper with numpy-aware batching and a verbose progress bar. concurrent.futures offers a ProcessPoolExecutor that shares its executor interface with ThreadPoolExecutor, which makes swapping between the two a one-line change when I/O-versus-CPU trade-offs shift at runtime. Dask scales the same idea across clusters and is the standard choice for Sydney-based data-engineering teams working with multi-terabyte datasets.
A quick checklist of when multiprocessing is the right tool:
- The task spends more than half its time in pure Python or numeric computation
- Inputs can be chunked into independent pieces with no shared mutable state
- The dataset fits in memory per worker, or can be streamed from disk
- You have at least two physical CPU cores available
- You have measured the serial version and confirmed it is too slow
Common mistakes that turn a parallel script into a slower serial one:
- Pickling a large object on every call instead of using shared memory
- Forgetting
if __name__ == "__main__":and spawning workers that recursively spawn more workers - Using more processes than physical cores and drowning in context-switch overhead
- Sharing a database connection across workers and tripping over thread-safety bugs
- Calling pool.map on a generator that produces millions of tiny items, drowning in IPC overhead
The pattern that works in production is unglamorous: a clean function, a fixed-size pool sized to the hardware, shared data loaded once, results collected once, and the whole thing wrapped in a context manager so workers exit cleanly even if the main process crashes. From a one-off script on an analyst's laptop to a scheduled batch job running on a 64-core instance in the AWS Sydney region, the same shape holds. The multiprocessing module rewards measured use, and once the mental model clicks, scripts that previously took a coffee break to finish start returning before the kettle boils.