Abstract flowing gradient in deep indigo and blue tones, smooth and luminous, evoking a modern digital learning atmosphere

Computer Science and programming articles. We do not sell courses.

Building a scalable data pipeline with Pandas and Apache Arrow

Every data engineer eventually meets the same wall: the notebook that ran fine on a 200-megabyte CSV now crawls when the file grows to 20 gigabytes. The cause is rarely Python itself and almost always the friction between in-memory data structures and the storage format on disk. Combining Pandas with Apache Arrow gives you a way to bridge that gap without rewriting your whole stack.

Pandas has been the lingua franca of tabular data analysis in Python for over a decade. Apache Arrow is a relatively recent cross-language standard for columnar memory. When you let Pandas operate on Arrow-backed data instead of its traditional NumPy arrays, you get faster I/O, cheaper inter-process communication, and a path to larger-than-RAM datasets.

In this walk-through you will see how to wire the two libraries together, where the speedups actually come from, and which pitfalls still bite teams in production. The examples assume Python 3.10 or newer and run on Linux, macOS or Windows without changes.

Australian data teams often hit the same bottlenecks a little earlier than their counterparts elsewhere because datasets arrive from sources spread across multiple time zones. Whether you are pulling weather observations from the Bureau of Meteorology in Melbourne, transaction logs from a fintech in Sydney, or agricultural sensor data collected on properties in regional Queensland, the diversity of formats demands a flexible pipeline. Tools that minimise data copies are particularly valuable when your compute budget is measured in dollars per hour and your connectivity occasionally drops out on a long-haul flight between Perth and Adelaide.

Installing Pandas and PyArrow in a reproducible environment

The simplest way to get started is pip install pandas pyarrow, but production pipelines deserve a pinned environment. Create a requirements.txt with exact versions such as pandas==2.2.2 and pyarrow==16.1.0, or better yet, a pyproject.toml for Poetry or uv. Both libraries move quickly, and Arrow in particular has had breaking changes around its C++ ABI in the past.

If you are working inside a managed platform such as AWS EMR, GCP Dataproc or an on-premises cluster run by a university in Melbourne or Brisbane, Arrow wheels may not match the system's glibc. Building PyArrow from source adds ten to fifteen minutes to image baking. Many Australian teams pre-build a Docker image on Amazon ECR or Google Artifact Registry; once it is in the registry, she will be right for the next six months.

A common gotcha is mixing Pandas releases with Arrow releases that pre-date or post-date the new PyCapsule protocol. Pandas 2.x uses the Arrow PyCapsule interface to hand tables to other libraries without copies. Pandas 1.x does not, so an older deployment will silently fall back to a slower path.

Reading source files into Arrow-backed tables

PyArrow's csv.read_csv and parquet.read_table return Arrow Table objects directly. Converting them to Pandas is as simple as df = table.to_pandas(), but that conversion is where many pipelines lose their edge. If you only need a few columns, filter or project before the conversion with table.select(["col_a", "col_b"]).

For partitioned Parquet datasets, pq.ParquetDataset lets you read across many files in a single call. A typical Australian use case is census data distributed by the Australian Bureau of Statistics, which ships a year of records across hundreds of small Parquet files. Reading them as a single logical table avoids the overhead of opening and closing files repeatedly.

When you must read CSVs that include Australian-specific quirks such as DD/MM/YYYY dates, currency strings with a leading dollar sign, or numbers written with a comma as the thousands separator, pass explicit converters or use pyarrow.csv.ConvertOptions to define column types up front. Inferring types on a 50-gigabyte file can consume gigabytes of RAM before the first row is returned.

Transforming data without losing performance

Once a table is in memory as an Arrow Table, every column operation can stay in Arrow until you actually need Pandas semantics. pc, the pyarrow compute module, covers most of what you would do with NumPy or Pandas: filtering with pc.equal, computing pc.add, or extracting parts of timestamps with pc.hour. The output of these functions is another Arrow Table, so chaining them does not allocate intermediate Pandas DataFrames.

When you eventually need Pandas, do the conversion once at the end of a chain. A pattern from Sydney transport feeds: load the day's tap-on records as Arrow, filter by mode of transport, aggregate by hour, then call .to_pandas() for matplotlib. The visualisation step tolerates the conversion cost because it happens only once.

If your pipeline feeds a downstream library that also supports Arrow, such as Polars, DuckDB or CuDF on GPUs, you can skip the Pandas conversion entirely. This is where the PyCapsule protocol shines: the same Arrow buffer is shared between processes with zero copy. A team in Perth working on mining drill-core data shipped a 40 percent latency improvement simply by replacing a to_pandas round-trip with a direct handoff to DuckDB.

When the data has natural graph structure, you will often want to combine tabular processing with a traversal step. Australian transport feeds, for example, can be held as Arrow tables while the underlying route network is walked separately. The a-tutorial-on-the-breadth-first-search-for-graph-traversal resource walks through one such hand-off in Python, showing how to keep the tabular joins in Pandas, the buffers in Arrow, and the graph traversal in its own dedicated loop.

Streaming and chunked processing for large datasets

Not every dataset fits in RAM. Pandas can read CSVs in chunks via the chunksize parameter, and PyArrow supports streaming reads with RecordBatchFileReader or ipc.open_stream. The trick is to keep each chunk small enough that the Python garbage collector can reclaim memory between batches.

A useful pattern is to read in batches of 100,000 rows, transform each batch into Arrow, write the result to a partitioned Parquet dataset on local NVMe or an S3 bucket in the ap-southeast-2 region, then discard the batch. This keeps peak memory close to the chunk size and gives you a fault-tolerant checkpoint. If you are running on a laptop on the train from Central Station to the Sydney Airport terminals, the same pattern works with the local file system.

Real-time ingestion is a different beast. When you need to process small, high-frequency events, even the overhead of constructing a Pandas index matters. Arrow's RecordBatch is the right unit of work here, and libraries such as pyarrow.flight let you stream batches over gRPC between services. Streaming systems that handle millions of small wagers, such as low-stakes wagering platforms processing cent-level bets, rely on this same low-latency transport to push events through without the per-record overhead of a DataFrame.

Memory profiling and zero-copy sharing

The phrase "zero copy" gets thrown around a lot in Arrow marketing material, but the effect is real. When two libraries in the same process share an Arrow buffer, neither makes a duplicate of the underlying memory. For a 1-gigabyte column, that is 1 gigabyte of RAM saved per hop. Across a pipeline with five hops, you can avoid 5 gigabytes of redundant allocation.

You can verify zero-copy behaviour with pyarrow.compute and the __array_interface__ attribute. If a Pandas Series backed by Arrow shares its underlying buffer with an Arrow ChunkedArray, the data pointer will match. Some operations such as astype with a different type do break sharing, and you will need to either accept the copy or restructure your pipeline to avoid the cast.

A common Australian scenario is processing satellite imagery over the country. Datasets from Geoscience Australia routinely exceed 100 gigabytes for a single scene. Reading them through rasterio into Arrow-compatible buffers, then slicing only the bands you need, keeps memory pressure manageable. Profilers such as memray or tracemalloc can be attached to a script to confirm that buffer sharing is actually happening and not just claimed in documentation.

Benchmarking and optimisation in practice

Before you optimise, measure. The pyarrow.benchmark module and the third-party pytest-benchmark plugin both let you record timings for individual operations. Keep the benchmark in version control so regressions show up in code review.

A reasonable baseline is to time three configurations: pure Pandas with NumPy, Pandas on Arrow, and pure Arrow with pc functions. For aggregations such as group-by on a categorical column, Arrow is usually faster because the columnar layout matches the access pattern. For row-wise operations that touch every column, Pandas can be competitive because NumPy arrays avoid some of Arrow's metadata overhead.

Hardware matters. An M3 MacBook in Brisbane gives different absolute numbers from a 64-core AMD EPYC in an AWS Sydney region. What you should compare is the ratio. If the Arrow pipeline is 2.1 times faster on your laptop, it will probably be 1.8 to 2.4 times faster in the cloud. Consistency is what you want, no dramas if the absolute numbers shift as long as the pattern holds.

When the ratio flattens, look for accidental copies. Typical culprits are calling to_pandas() inside a loop instead of after it, or using apply with a Python function on every row of a million-row DataFrame. Vectorised pc calls almost always win. If you must iterate, do it over an Arrow RecordBatch and call to_pylist() once at the end.

Practical recommendations for your next pipeline