Building Mini Spark, a Tiny Distributed Computing Engine · Growing BitsGrowing Bits
Recent Notes<br>Building Mini Spark, a Tiny Distributed Computing Engine
Aug 21, 2026
A Mathematical Theory of the Antilibrary
Aug 06, 2026
Ingesting Webhooks with AWS Serverless
Aug 05, 2026
Building the Pointer Architecture with Terraform
Aug 04, 2026
SearchSearch
Dark modeLight mode<br>Reader mode
Building Mini Spark, a Tiny Distributed Computing Engine<br>Garden statusPage typeGuide<br>Growth stageSeed<br>Version0.1<br>Aug 21, 202621 min read<br>distributed-computing<br>data-engineering<br>mini-spark
Building Mini Spark, a Tiny Distributed Computing Engine
I wanted to understand what actually happens inside a distributed data engine, so I built a very small one in Python.
The result is Mini Spark. The name is intentionally ambitious: this first version is really a small MapReduce executor. It splits input into tasks, runs those tasks on several workers, shuffles intermediate results over HTTP, and retries work when a worker disappears. We will use pizza orders as the dataset because distributed systems are complicated enough without a complicated example too.
The complete source code is available in the Mini Spark repository.
A real-world analogy
Imagine you are organising a huge pizza party. Thousands of people have sent you their orders, and you have ended up with a CSV containing a name and a pizza type:
NamePizza typeAdaMargheritaGraceMushroomsLinusMargherita
You need one total for each pizza type. On your own, you would read the file from top to bottom and keep a running count. With Alice, Bob, and Chris helping, you can split the list into three chunks and give one chunk to each person. They count their chunk, send back the totals, and you add those totals together.
That is the basic shape of the engine we are going to build.
Meet the compute engine
You play the coordinator (often called the driver), while Alice, Bob, and Chris are the workers . The coordinator splits up the job, hands out work, keeps track of progress, and decides what to do when something fails.
The workers will not always behave nicely. One may be slow, one may crash halfway through a task, and another may finish after the coordinator has already given up on it. Dealing with those cases is a large part of distributed execution.
Before getting into failures, it helps to separate the computation we asked for from the work the cluster has to perform.
The logical plan: what needs to happen
At the logical level, the pizza query is just:
Read every order.
Group the orders by pizza type.
Count the orders in each group.
There are no workers or partition sizes in that description. The query stays the same whether it runs on one worker or a thousand.
The physical plan: how it will happen
The physical side is more practical. It has to decide how that query will actually run:
Divide the list into smaller pieces.
Create tasks that process those pieces.
Assign the tasks to available workers.
Move partial counts for the same pizza type to the same place.
Combine them into the final totals.
This first version does not have a real logical-plan API or planner yet. We build this physical plan directly in CountPizza(). Part two will separate those layers properly, but the distinction is still useful for understanding the execution model here.
Tasks and partitions
Two ideas do most of the work: partitions split the data, and tasks describe what to do with each piece.
Partitions divide the data
A partition is simply a chunk of a dataset. Instead of treating the order list as one giant file, we divide it into ranges:
Pizza orders<br>├── Partition 1: rows 1–100<br>├── Partition 2: rows 101–200<br>├── Partition 3: rows 201–300<br>└── ...
Alice can work on one partition while Bob and Chris work on others. The partition size is a trade-off: a few large partitions are cheap to coordinate but harder to balance and expensive to retry; lots of tiny partitions spread out nicely but create more scheduling overhead.
Tasks divide the computation
A task is the piece of work we can hand to a worker. For Mini Spark, the useful mental model is:
Task = operation + input partition
Or, if you prefer notation:
T = (f, P)
where f is the operation and P is the input partition. Executing the task produces a result:
TaskResult = f(P)
Real engines often pipeline several operations into one task, but this simpler definition gives us something concrete to schedule, watch, and retry.
For example:
Task 17 = count pizza types in partition 17
A task is not permanently tied to a worker. Alice, Bob, or Chris should all be able to run task 17 and get the same answer. If Alice disappears, the coordinator can give the same task to Bob.
So we have three different things:
Partition = a unit of data<br>Task = a unit of work<br>Worker = something that executes the work
A worker picks up a task, reads the task’s input partition, runs the operation, and stores the result. Another task may then...