A Python ring buffer for continuous streaming data

A ring buffer, sometimes called a circular buffer or circular queue, is a fixed-size data structure that wraps around on itself once it reaches the end of its allocated memory. It behaves like a First-In-First-Out pipeline where the oldest record is automatically overwritten by the newest one. For anyone handling telemetry from sensors, market data ticks, or live log streams, this overwrite-on-full behaviour is often exactly what you want: keep the last N samples, throw the rest away, and never let memory balloon out of control.

In Australian contexts, ring buffers quietly underpin plenty of systems. Market data feeds from the ASX stream through them, regional water-management sensors in the Murray-Darling basin use them to keep recent readings in RAM, and small Python services in Perth mining dashboards rely on them to smooth out bursty network traffic. This article walks through the design, implementation, complexity profile, and a few practical extensions of a ring buffer you can drop into a Python streaming project today.

What a ring buffer actually is

At its core, a ring buffer is just an array plus two indices. One index marks where the next write should occur, the other marks where the next read should occur. When either index reaches the end of the array, it loops back to zero, which is why the structure looks like a ring when you draw it. The "is empty" versus "is full" distinction is usually handled with a counter or a sentinel slot.

This structure is widely used in kernel networking stacks, audio processing, and embedded firmware because it gives you constant-time enqueue and dequeue operations without pointer juggling. The Python standard library's collections.deque is thread-safe and convenient, but it does not let you overwrite the oldest entry automatically when it fills up. That difference matters with streaming data: you frequently want a moving window of the most recent observations rather than an ever-growing queue that will eventually exhaust memory.

Compared to a naive list, a ring buffer wins on speed and memory predictability. Appending to a list is amortised O(1) in CPython, but resizing happens behind the scenes, and pop(0) is O(n) because every element has to shift. A ring buffer sidesteps both issues by treating storage as a fixed slab. Compared to a binary heap, a ring buffer is simpler and faster for plain FIFO traffic, as explained in the walkthrough on implementing a binary heap for heapsort in Python, where the priority semantics and extra bookkeeping are discussed in detail.

Choosing the right building blocks in Python

The implementation below uses a plain list as the underlying storage. You might be tempted to reach for array.array or numpy.ndarray for performance, and those are reasonable choices if your records are numeric. For a general-purpose buffer that holds Python objects, a list is the most readable starting point and is plenty fast for most streaming workloads.

You also need a strategy for distinguishing empty from full. The classic approaches are: keep an explicit count, leave one slot unused, or use a sentinel value. Keeping an explicit count is the easiest to reason about and is the approach used here. It costs one extra integer per buffer, which is negligible compared to the buffer slots themselves.

Capacity selection is another design knob. Size too small and you lose old data; size too large and you waste memory and increase cache misses. In practice, you want a capacity that comfortably covers a few seconds to a few minutes of peak traffic. For a service in Sydney processing smart-meter readings every second from a few thousand endpoints, a capacity of a few thousand slots per tenant is a sensible starting point.

Key design decisions to lock in early:

• Whether to allow silent overwrite on overflow or to raise an exception
• The choice between a typed array.array and a plain list for storage
• Whether to expose a thread-safe API or leave concurrency to the caller
• How to represent the empty-versus-full state, since both look identical with raw indices
• The expected peak rate of incoming items, which drives the capacity

Putting the code together

Below is a small, self-contained implementation. It is deliberately kept under fifty lines so the structure is easy to read and adapt.

class RingBuffer:
    def __init__(self, capacity: int):
        if capacity <= 0:
            raise ValueError("capacity must be positive")
        self._data = [None] * capacity
        self._capacity = capacity
        self._head = 0  # next write index
        self._tail = 0  # next read index
        self._size = 0

    def push(self, item) -> None:
        self._data[self._head] = item
        self._head = (self._head + 1) % self._capacity
        if self._size < self._capacity:
            self._size += 1
        else:
            self._tail = (self._tail + 1) % self._capacity

    def pop(self):
        if self._size == 0:
            raise IndexError("pop from empty buffer")
        item = self._data[self._tail]
        self._tail = (self._tail + 1) % self._capacity
        self._size -= 1
        return item

    def __len__(self) -> int:
        return self._size

    def __iter__(self):
        idx = self._tail
        for _ in range(self._size):
            yield self._data[idx]
            idx = (idx + 1) % self._capacity

The modulo arithmetic is the entire trick. Every time an index advances, you take it modulo the capacity, which causes the index to wrap around to zero when it reaches the end of the underlying list. The iterator yields elements in insertion order, starting from the oldest.

A useful variant is the overwrite semantic. When the buffer is full and a new item arrives, the push method silently overwrites the oldest record by advancing the tail pointer. This is what makes the data structure suitable for streaming data, where losing the oldest sample is preferable to blocking the producer.

Complexity profile and memory footprint

All of the core operations run in constant time. push is O(1), pop is O(1), __len__ is O(1), and __iter__ is O(n) over the current contents but does not allocate a separate copy. This is the main reason ring buffers are so popular in latency-sensitive paths, including audio engines in video-conferencing tools and log shippers used in fintech firms operating out of Melbourne.

Memory usage is exactly capacity slots, regardless of fullness. There is no resizing and no per-element overhead beyond the Python object reference in the list. If you switch to array.array with a typed code such as 'd' for double-precision floats, you get a footprint close to capacity * 8 bytes, which is hard to beat without dropping into a C extension.

This predictability matters on a constrained host. A small edge gateway installed on a remote cattle station in northern Queensland may only have a few hundred megabytes of RAM, and you do not want a queue that grows without bound while waiting for an upstream network to recover. The fixed footprint of a ring buffer turns that risk into a non-issue.

Thread safety and the GIL

CPython's global interpreter lock means that single operations on built-in types are atomic, but compound operations are not. A push followed by a pop from different threads can interleave in ways that produce surprising results. For most streaming use cases, the safe pattern is to wrap access in a threading.Lock or use a queue.Queue if you genuinely need a multi-producer, multi-consumer setup.

If you are processing data for machine-learning pipelines, the choice of buffer can interact with batching in subtle ways. The discussion of weighted sampling and ordering trade-offs in how to solve the subset sum problem with dynamic programming touches on similar "which items do I keep" questions, and the same instinct applies here: pick the buffer size and eviction policy that match the statistical properties of the data you care about.

For single-threaded asyncio code, the buffer is usually accessed by a single coroutine at a time and the lock is unnecessary. For multiprocessing pools, you would need to share the buffer through shared memory or move it to a sidecar service, since processes do not share Python objects by default.

Real-world streaming scenarios in Australia

Ring buffers turn up in surprisingly many places. ASX-style market data handlers keep the most recent ticks per symbol so that downstream models can compute rolling features without querying a database. Log shippers in Australian SaaS companies use them to buffer records when the destination is briefly unavailable, then flush once the destination comes back. Even a weather-station hobbyist in Adelaide pulling data from a Bureau of Meteorology feed can use one to maintain a rolling average of the last hour of readings.

In machine-learning pipelines, a ring buffer is often the missing piece between a raw stream and a feature extractor. A classifier that needs the last 60 samples to make a prediction is a perfect fit: a ring buffer of size 60 lets you pop the oldest sample, push the newest, and hand the contents to the model in O(60) time, which is effectively constant. The trade-offs between online and batch learners, which the article on understanding the AdaBoost algorithm for boosting classifiers explores in some depth, become much more concrete once you have a steady source of buffered examples feeding the learner.

A lot of regional IoT work also depends on this pattern. Soil-moisture deployments across vineyards in South Australia, weather stations feeding the CSIRO, and pipeline monitors in the Pilbara all benefit from a bounded in-memory buffer that survives short network outages without losing recent history.

Validating and extending the buffer

A good way to test a ring buffer is to push more items than the capacity, then iterate and check that exactly the last capacity items come back in the right order. Another useful test is to interleave pushes and pops to confirm that head and tail wrap correctly across the boundary between the end and the start of the underlying list. A property-based test using hypothesis is a clean way to generate these sequences automatically and catch off-by-one bugs that are easy to miss in hand-written tests.

Useful extensions to consider for production:

• A fixed-capacity variant of numpy.ndarray for numeric streams
• A peek_last(n) method that returns the most recent N items without removing them
• A clear() method that resets the indices and counter in one step
• A serialisation helper that writes the current contents to disk for debugging
• A thread-safe wrapper built on collections.deque plus a threading.Lock
• Snapshot-and-resume support so a buffer can persist across a service restart

For high-throughput services in Brisbane's resource-tech sector, the most valuable extension is usually the thread-safe wrapper, which keeps the API familiar while adding the concurrency guarantees that a production deployment requires. Combined with structured logging and a few health-check counters, the simple class above becomes a solid building block for a wide range of streaming systems in Australia and beyond.