Modest contributions to the "big data" ecosystem | Guy Waldman<br>New
go|<br>rust|<br>bigdata<br>Modest contributions to the "big data" ecosystem
Guy Waldman
August 3, 2026 (1 hour ago)
As someone who's no stranger to high-scale ETL (Extract, Transform, Load) workloads that work across huge data lakes, this is a love-hate relationship (mostly love though).<br>Having said that, I've recently been spending some time around Polars and the broader DataFrame ecosystem, mostly because I keep running into the same boundary: the data is messy, but the useful part of the system wants a schema.<br>So, introducing two side projects around that problem space:
polars-fastjson, a lenient, schema-aware JSON projection for Python Polars.
golars, a Polars-like, Arrow-native DataFrame API for Go.
Neither is intended to replace Polars core, they are supplemental additions to the already thriving polars ecosystem. Standing on the shoulders of giants, as they say.
##<br>polars-fastjson
Here's something I was very surprised to see:<br>import polars as pl
data = [<br>{"id": 1, "metadata": '{"role": "admin", "status": "active"}'},<br>{"id": 2, "metadata": '{"role": "user", "status": "inactive"}'},<br>{"id": 3, "metadata": '{"role": "guest", "status": "active"'}, # Malformed JSON (missing })<br>{"id": 4, "metadata": '{"role": "user", "status": "active"}'}
df = pl.DataFrame(data)<br>parsed_df = df.with_columns(<br>pl.col("metadata").str.json_decode(pl.Struct).alias("parsed_meta")<br>).show()PYTHON
This raises an exception:<br>ComputeError: error deserializing JSON: json parsing error: 'ExpectedObjectKey at character 115 ('{')'
This error occurred in the following expression:<br>col("metadata").str.json_decode()ERROR
This even happens when the JSON is well formed, but the schema changes between rows:<br>import polars as pl
data = [<br>{"id": 1, "metadata": '{"role": "admin", "status": "active"}'},<br>"id": 2,<br>"metadata": '{"role": "user", "status": [42]}',<br>}, # Malformed data (status should be a string)<br>{"id": 3, "metadata": '{"role": "guest", "status": "active"}'},<br>{"id": 4, "metadata": '{"role": "user", "status": "active"}'},
df = pl.DataFrame(data)<br>parsed_df = df.with_columns(<br>pl.col("metadata")<br>.str.json_decode(<br># We even specify a schema here<br>pl.Struct([pl.Field("role", pl.String), pl.Field("status", pl.String)])<br>.alias("parsed_meta")<br>).show()PYTHON
Also blows up:<br>ComputeError: error deserializing JSON: error deserializing value "Array([Static(U64(42))])" as string.
Try increasing `infer_schema_length` or specifying a schema.
This error occurred in the following expression:<br>col("metadata").str.json_decode()ERROR
So the problem statement is - we have many rows and we want to decode JSON efficiently and also in such a way that we can be lenient about parse errors (and understand why they happened).<br>Alternatively, we could use json_path_match (for example, pl.col("raw_json").str.json_path_match("$.user.name")) but this is highly inefficient, since you need to reparse the same JSON row if you want to extract more than 1 field.<br>Here is where I scoured the internet to see what I was missing, since this doesn't seem like that much of a niche problem. But it looks like there's no popular solution here (happy to hear if I'm wrong, please DM me!).<br>So I (+ some AI agents) wrote polars-fastjson.<br>It takes a different approach: provide the target schema once, then project each JSON string into a typed Struct and allow for leniency (don't simply raise an exception if something goes wrong).<br>import polars as pl<br>from polars_fastjson import fastjson_decode
schema = {<br>"id": pl.String,<br>"score": pl.Float64,<br>"tags": pl.List(pl.String),
parsed = df.with_columns(<br>fastjson_decode(pl.col("payload"), schema=schema).alias("parsed")<br>)PYTHON
By default:
Malformed rows become null structs
A bad "leaf" field becomes null and valid sibling fields are retained
Compatible values can be coerced (my personal choice here, can be configured)
In addition:
Strict modes are available when a pipeline should fail instead
Nested structs and lists are supported
You can supply a schema by a Polars dtype, a dictionary, a dataclass, a TypedDict, or a Pydantic model.
You can ask to emit diagnostics which allows for a summary to understand why rows were nulled without logging each one (huge I/O strain, and also noisy)
The performance has been promising in local benchmarks, including at million-row scale, and appears to scale roughly linearly with the number of rows. The benchmark is included in the repository if you want to try it yourself.
##<br>golars
The motivation for golars is slightly different from polars-fastjson.
I love Polars and love Go. For some experiments, I wanted to stay in Go while still using Polars' lazy query model and native execution, but couldn't find a well-supported binding that gave me that combination.<br>Generally, I wanted to explore what a DataFrame API could feel like from Go without reimplementing a relational engine in Go or requiring Python as the host language.<br>The project...