Abstract flowing gradient in deep indigo and blue tones, smooth and luminous, evoking a modern digital learning atmosphere

Computer Science and programming articles. We do not sell courses.

Parallel work with Python multiprocessing pools

Python's multiprocessing.Pool runs independent jobs in separate worker processes, allowing a program to use several CPU cores instead of handling every item in one process. It is especially useful for embarrassingly parallel work: independent records that can be processed in any order and combined afterwards.

The gain is not automatic. Process creation, argument and result serialisation, and scheduling all cost time, while map waits for the complete result set. CPU-bound Python code benefits most; network or disk waits often work better with threads, and very small inputs can run faster sequentially.

Australian teams encounter this pattern in familiar settings. A Melbourne retailer might score loyalty transactions, a Sydney logistics business might geocode delivery routes, and a Brisbane environmental project might resample climate records. Data tied to Australian energy prices, postcode boundaries or ASX activity often starts in modest CSV extracts and later outgrows a laptop.

Running the same workload in Australian cloud regions adds data-residency and network-latency choices, so portability matters as much as speed. A dependable implementation therefore needs more than a pool constructor. You must choose sensible tasks, control the amount of data crossing process boundaries, measure the speed-up on representative inputs, and make failures visible. The examples below focus on practical CPU parallelism while remaining portable across Linux, macOS and Windows.

Choosing the right execution model

Match the execution model to the bottleneck rather than treating every slow function as a candidate for multiprocessing. The comparison below separates lightweight approaches from process-based ones and highlights the cost each introduces.

Model Best fit Main advantage Main limitation Result order
Sequential code Small jobs and simple flows Lowest overhead Blocks other work Input order
ThreadPoolExecutor Network and file I/O Shared memory and simple cancellation GIL limits Python CPU work Input order
multiprocessing.Pool Batched CPU-bound tasks Straightforward parallel map Process and pickle overhead map preserves order
ProcessPoolExecutor Fine-grained or asynchronous tasks Modern API and richer features Same process costs Input order

Python's Global Interpreter Lock allows only one thread to execute Python bytecode at a time within an interpreter. Separate worker processes get separate interpreters, so CPU-heavy Python can run on multiple cores. Some native libraries release the GIL, which makes a thread benchmark essential before choosing a more complex design. Pool remains convenient for bulk maps, although ProcessPoolExecutor may suit new code needing a more flexible executor interface.

Creating a safe first pool

A minimal pool should have a top-level worker function and a protected entry point. The guard is essential when the start method is spawn, as used by default on macOS and Windows, or when using forkserver.

from multiprocessing import Pool

def square(value):
    return value * value

if __name__ == "__main__":
    with Pool(processes=4) as pool:
        results = pool.map(square, range(20), chunksize=2)

    print(results)

The entry-point guard prevents child processes from recursively creating workers when they re-import the module. It matters under spawn and forkserver, and remains harmless under fork. Pool normally uses all available CPUs, yet that default can overcommit memory or ignore container quotas, so pass an explicit processes value after measuring the host. For a graceful asynchronous workflow, call close() after the final submission and join() after consuming the results; the context manager's exit path terminates workers.

Choosing how tasks are scheduled

The standard map call is blocking and returns results in input order. For long jobs, imap_unordered can begin yielding completed values earlier, which is valuable when downstream work only needs each answer as it arrives.

from multiprocessing import Pool

def score(record):
    return record["price"] * record["quantity"]

if __name__ == "__main__":
    with Pool(4) as pool:
        for total in pool.imap_unordered(score, records, chunksize=100):
            print(total)

Use imap when callers require input order, and imap_unordered when completion order is acceptable. Ordering guarantees can add waiting: a slow first record holds back later results even if the remaining workers have finished. An ordered stream is still useful when outputs correspond directly to source rows.

chunksize is the main tuning knob. Larger batches reduce scheduling overhead but can leave workers idle when task durations vary; smaller batches balance better but increase communication. Benchmark with real inputs because a collection of tiny records and a list of large matrices have very different costs. Avoid one task per element when each element takes only a few microseconds.

Making data safe to send

Arguments are pickled on the way to a worker and results are pickled on the way back. Worker functions must therefore be importable at module level, while arguments and return values need serialisable contents. Lambdas, open file handles, database cursors and many local objects cannot cross the boundary directly. Since unpickling can execute reconstruction instructions, keep exchanged values within a trusted boundary.

Globals are not shared live state. A convenient pattern is to initialise read-only worker configuration once with initializer and initargs, then reference it inside each task:

MODEL = None

def init_worker(model):
    global MODEL
    MODEL = model

def predict(record):
    return MODEL.predict([record])

if __name__ == "__main__":
    with Pool(4, initializer=init_worker, initargs=(model,)) as pool:
        predictions = pool.map(predict, records)

Large collections deserve special care because Pool.map pickles every argument and materialises every result. Batch several records per task, share read-only arrays with multiprocessing.shared_memory or memory maps, and merge partial results in the parent. Mutations in one worker do not appear in another. If a task fails, catch the exception inside the worker and return a structured error, since map otherwise re-raises it when the parent consumes that result.

Applying parallelism to real analyses

Text analysis is a good candidate when files or documents are independent. Developers adapting the Boyer-Moore string-search guide can split a corpus by document and let workers scan separate files. The parent can then combine counts, rankings or match positions without sharing mutable parser state.

Image processing, geospatial calculations and feature extraction often show the same shape. For geospatial work, partition by bounding box or spatial tile so that each record has a predictable location; avoid sending a separate database query for every tiny task because connection overhead can dominate the computation.

The Australian examples map neatly to this model. A Brisbane team can distribute climate-station transformations across workers, while a Sydney operations team can calculate route features for delivery batches. Keep source extraction and final aggregation in the parent process so validation rules remain consistent, and record the worker count and input partition beside each result for reproducibility.

Benchmarking before scaling

Measure before and after, using the same data and a single run for development. Wall-clock time, throughput and peak memory tell different stories: a pool may reduce runtime while increasing RAM use. time.perf_counter() is useful for elapsed time, and a profiler can show whether workers spend their time in Python arithmetic, native numerical code or I/O. No pool can remove serial stages, so Amdahl's law sets a practical ceiling on speed-up.

For dimensional reduction, the principal component analysis primer helps separate fitting the transformation from projecting records. Fit the model once on the training data, then parallelise projection only if profiling shows it is worthwhile. Local cloud choice also affects results: benchmark comparable instances in Australian Sydney and Melbourne regions, confirm data-residency requirements, and compare the complete AUD cost rather than relying on the cheapest advertised hourly rate.

Sweep a small range of worker counts rather than assuming every core is useful. Numerical libraries can start their own threads inside each process, causing oversubscription; limiting BLAS or OpenMP threads to one often improves a CPU pool. On machines with expensive electricity or constrained cooling, fewer workers may also be the economical choice. For variable-duration jobs, set maxtasksperchild when long-lived workers accumulate memory, and verify that recycling does not erase useful state.

Operating patterns and failure responses

A short release checklist catches common deployment problems:

These habits suit the practical, reproducible examples valued by maintainers and attendees at Australian events such as PyCon AU. When a run behaves unexpectedly, match the symptom to a likely cause:

The central principle is to give each worker a substantial, independent unit of work, minimise serialisation and treat measurement as part of the implementation. A pool is effective when those conditions hold, and unnecessary when a sequential loop, vectorised library or thread-based I/O path is already faster.