Pydantic Data Validation in Python

When you build a Python service that ingests records from an API, a spreadsheet, or a Kafka topic, one quiet mistake can poison an entire dataset. A string that should be an integer arrives as "42", a date slips in as "next Tuesday", and your downstream pipeline silently miscalculates revenue. Engineers in Sydney and Melbourne fintechs encounter this constantly when reconciling transactions against the ASX feed, where a missing decimal can shift a balance by orders of magnitude. Pydantic is a library designed to catch exactly these slips at the boundary of your system, turning fuzzy real-world input into rigorously typed Python objects before that data ever reaches your business logic.

The library leans on Python's native type hints, which means your validation rules double as documentation. Anyone reading your codebase can immediately see the expected shape of a request, a row, or a configuration file. For developers coming from a background in scientific computing or statistical modelling, the experience feels familiar: you declare a schema, you run the data through it, and you either get a clean object back or an actionable error. That blend of clarity and rigour has made Pydantic the standard choice for new Python services across Australian teams, from Perth mining consultancies to Sydney product startups.

Installing Pydantic and Choosing a Version

Pydantic is available on PyPI, so a single pip install pydantic will get you running within seconds. If you are working in a corporate environment such as a Brisbane bank or a Canberra government department, you may want to pin the version in your requirements.txt or pyproject.toml because major releases occasionally introduce breaking changes around field aliasing and strict mode. Most teams in Australia still rely on Pydantic v2, which moved validation to a Rust core for substantially faster throughput than v1.

Before you commit to a version, take a moment to consider what your validation workload actually looks like. If your service is processing thousands of Medicare claim records per minute, performance matters. If you are validating configuration for a small internal tool used by a marketing team in Adelaide, readability matters more. Both versions handle the same schemas, but their performance profiles and a handful of API differences will influence which one fits your context. You can inspect the latest release notes on PyPI and pick the one that aligns with your runtime constraints.

Crafting Your First Schema

A Pydantic model is a regular Python class that inherits from BaseModel. Fields are declared with type annotations, and Pydantic uses those annotations to coerce and validate the incoming data automatically. Consider a simple example for an e-commerce order:

from pydantic import BaseModel

class Order(BaseModel):
    order_id: int
    customer_email: str
    total_aud: float
    is_paid: bool

If someone passes total_aud="199.95", Pydantic will coerce the string into a float. If they pass is_paid="yes", however, the model will raise a ValidationError, because the library refuses to guess what a non-boolean string means. This is the sweet spot: Pydantic is generous with safe coercions (strings to numbers, ISO timestamps to datetimes) and strict about anything ambiguous. For Australian readers, this matters when ingesting ATO reporting data, where a True/False flag for GST registration must never be silently coerced from "y".

You can also make fields optional by using Optional[float] or the newer float | None syntax, and you can provide defaults. Optional fields are perfect for tracking attributes that may or may not arrive, such as a customer's middle name or a delivery driver's phone number. The model behaves like a dataclass with superpowers, giving you attribute access, a helpful __repr__, and a .model_dump() method that returns a plain dictionary for serialisation back to JSON or a database row.

Custom Validators and Field Constraints

Type hints cover a lot of ground, but real-world data rarely stays inside tidy boundaries. Pydantic gives you two complementary tools for tighter rules: constraints declared on the field itself, and custom validators written as methods. Constraints use Field() and include things like min_length, max_length, ge (greater than or equal), le (less than or equal), and pattern for regex checks. The approach resembles how support vector machines carve clean boundaries through messy feature spaces; you decide what counts as valid here too, and the framework enforces it consistently.

For example, you could enforce that an Australian post code is exactly four digits:

from pydantic import BaseModel, Field

class Address(BaseModel):
    street: str
    suburb: str
    postcode: str = Field(pattern=r"^\d{4}$")
    state: str

When the rules are more elaborate, custom validators step in. You decorate a method with @field_validator and Pydantic calls it after type coercion. This is the right place to enforce business rules that the type system cannot express, such as rejecting a flight booking whose departure date is in the past, or normalising a phone number to E.164 format. Public-sector teams in Hobart and Darwin often write validators that map local government area names onto a canonical set, since raw submissions tend to mix abbreviations and full names.

For multi-field rules, @model_validator(mode="after") lets you inspect the entire model after every field has been validated. That is the layer where you compare a start date against an end date, or check that a discount percentage never exceeds the total order amount. Validation errors raised inside these methods bubble up as structured ValidationError instances, with a JSON payload that pinpoints exactly which field failed and why, which makes debugging far easier than chasing a generic ValueError.

Nested Models, Lists, and Discriminated Unions

Real data structures are rarely flat. An order contains line items, a customer contains multiple addresses, a sensor reading contains a list of measurements. Pydantic handles nesting naturally: you simply declare a field whose type is another BaseModel, and the library recurses into it. Lists are typed with list[Item], dictionaries with dict[str, Value], and tuples when the order of fields carries meaning.

Discriminated unions are especially handy when you ingest data from multiple sources with different shapes. Imagine a payments endpoint that can return either a credit-card payment or a bank-transfer payment. By adding a kind field and using Field(discriminator="kind"), Pydantic will pick the right subclass based on that tag, sparing you from writing a chain of isinstance checks. The same appetite for clear structure shows up across data work, and this practical introduction to Bayesian statistics demonstrates the same instinct for declaring assumptions explicitly before the analysis begins.

Configuration management is another area where nested models shine. You can model your entire application config as a tree of BaseModel classes, load the tree from environment variables or a YAML file, and rely on Pydantic to fail loudly if a required secret is missing. For a deployment running across multiple AWS regions from a Sydney operations team, that kind of fail-fast behaviour at startup is worth a great deal of late-night debugging.

Performance, Serialisation, and Integration Patterns

Once your schemas are in place, you will want them to stay fast. Pydantic v2's Rust core can validate millions of simple objects per second on a modern laptop, and it uses caching aggressively so repeated validations of identical objects cost almost nothing. The main lever you control is whether you reach for .model_validate() (which builds the object) versus .model_validate_json() (which parses JSON directly without an intermediate dict). For high-throughput services such as a payments gateway in Melbourne's fintech district, the JSON path is meaningfully faster and skips a redundant parsing step.

Serialisation follows the same naming. .model_dump() returns a Python dict, .model_dump_json() returns a JSON string, and both accept an exclude argument so you can omit sensitive fields such as tax file numbers when logging. When you combine this with FastAPI, Pydantic models become the request and response schemas automatically, and the framework produces OpenAPI documentation that reflects your constraints out of the box. That integration has made Pydantic the de facto choice for new Python web services, particularly across Australian startups that favour Python for both data engineering and API layers.

A few habits will keep your schemas healthy over time. Keep models small and composable rather than building one giant class that does everything. Centralise shared enums (such as the set of Australian states and territories, or supported currency codes) so they are consistent across services. Add unit tests for the failure cases as well as the happy path, because the value of Pydantic is precisely the errors it raises when reality does not match your assumptions. Treat your schemas as part of the contract between teams, version them deliberately, and your data boundary will hold firm no matter how chaotic the upstream inputs become.