Deciphering Glyph ::<br>Opaque Types in Python
Deciphering
Glyph
About
Archives
Mastodon
GitHub
Patrons
Opaque Types in Python
A proposed technique for exposing an opaque data structure with<br>idiomatic modern Python.
pythonprogramming Thursday May 21, 2026
Let’s say you’re writing a Python library.
In this library, you have some collection of state that represents “options” or<br>“configuration” for a bunch of operations. Such a set of options is a bundle<br>of potentially ever-increasing complexity. Thus, you will want it to have an<br>extremely minimal compatibility surface, with a very carefully chosen public<br>interface, that is either small, or perhaps nothing at all. Such an object<br>conveys state and might have some private behavior, but all you want consumers<br>to be able to do is build it in very constrained, specific ways, and then pass<br>it along as a parameter to your own APIs.
By way of example, imagine that you’re wrapping a library that handles shipping<br>physical packages.
There are a zillion ways to do it ship a package. There are different carriers<br>who can ship it for you. There’s air freight, and ground freight, and sea<br>freight. There’s overnight shipping. There’s the option to require a<br>signature. There’s package tracking and certified mail. Suffice it to say,<br>lots of stuff.
If you are starting out to implement such a library, you might need an object<br>called something like ShippingOptions that encapsulates some of this. At the<br>core of your library you might have a function like this:
async def shipPackage(<br>how: ShippingOptions,<br>where: Address,<br>) -> ShippingStatus:<br>...
If you are starting out implementing such a library, you know that you’re<br>going to get the initial implementation of ShippingOptions wrong; or, at the<br>very least, if not “wrong”, then “incomplete”. You should not want to commit<br>to an expansive public API with a ton of different attributes until you really<br>understand the problem domain pretty well.
Yet, ShippingOptions is absolutely vital to the rest of your library. You’ll<br>need to construct it and pass it to various methods like estimateShippingCost<br>and shipPackage. So you’re not going to want a ton of complexity and churn<br>as you evolve it to be more complex.
Worse yet, this object has to hold a ton of state. It’s got attributes, maybe<br>even quite complex internal attributes that relate to different shipping<br>services.
Right now, today, you need to add something so you can have “no rush”,<br>“standard” and “expedited” options. You can’t just put off implementing that<br>indefinitely until you can come up with the perfect shape. What to do?
The tool you want here is the opaque data type design pattern. C is lousy<br>with such things (FILE, pthread_*_t, fd_set, etc). A typedef in a<br>header file can easily achieve this.
But in Python, if you expose a dataclass — or any class, really — even if<br>you keep all your fields private, the constructor is still, inherently,<br>public. You can make it raise an exception or something, but your type checker<br>still won’t help your users; it’ll still look like it’s a normal class.
Luckily, Python typing provides a tool for this:<br>typing.NewType.
Let’s review our requirements:
We need a type that our client code can use in its type annotations; it<br>needs to be public.
They need to be able to consruct it somehow, even if they shouldn’t be<br>able to see its attributes or its internal constructor arguments.
To express high-level things (like “ship fast”) that should stay supported<br>as we add more nuanced and complex configurations in the future (like “ship<br>with the fastest possible option provided by the lowest-cost carrier that<br>supports signature verification”).
In order to solve these problems respectively, we will use:
a public NewType, which gives us our public name...
which wraps a private class with entirely private attributes, to give us<br>an actual data structure, while not exposing the constructor,
a set of public constructor functions, which returns our NewType.
When we put that all together, it looks like this:
10<br>11<br>12<br>13<br>14<br>15<br>16<br>17<br>from dataclasses import dataclass<br>from typing import Literal, NewType
@dataclass<br>class _RealShipOpts:<br>_speed: Literal["fast", "normal", "slow"]
ShippingOptions = NewType("ShippingOptions", _RealShipOpts)
def shipFast() -> ShippingOptions:<br>return ShippingOptions(_RealShipOpts("fast"))
def shipNormal() -> ShippingOptions:<br>return ShippingOptions(_RealShipOpts("normal"))
def shipSlow() -> ShippingOptions:<br>return ShippingOptions(_RealShipOpts("slow"))
As a snapshot in time, this is not all that interesting; we could have just<br>exposed _RealShipOpts as a public class and saved ourselves some time. The<br>fact that this exposes a constructor that takes a string is not a big deal for<br>the present moment. For an initial quick and dirty implementation, we can just<br>do checks like if options._speed == "fast" in our shipping and estimation<br>code.
However, the main thing we are doing here is preserving our flexibility to<br>evolve...