What a Data Engineer Misses About Transformers (Until You Build One) | Saran Teja Mallela before<br>the browser paints. Runs synchronously, no defer/async.<br>--><br>Skip to content<br>Go back<br>What a Data Engineer Misses About Transformers (Until You Build One)<br>3 Jul, 2026
I build data pipelines for a living. Kafka, Spark, Azure, the usual. I assumed that gave me decent intuition for how LLMs get trained. It is all just moving tensors through a DAG, right?
So over eight days in late June I worked through Karpathy’s Zero to Hero, typing every line: an autograd engine, bigram and trigram language models, an MLP with batch norm, and a GPT built up from a single attention head. I watched all ten videos and built through the GPT one. The code, commit history and all, is on GitHub.
My data engineering instincts were not missing details. They were pointed in the wrong direction, in five specific places. And then there were the bugs, which deserve their own section, because none of them crashed.
Table of contents
Open Table of contents
1. Backprop is lineage tracking that carries blame
2. The loss function is a data quality metric
3. Batch size is a hyperparameter, and batch norm couples your rows
4. Attention is a differentiable JOIN
5. Deep learning bugs don’t crash. They converge.
The one I haven’t built yet
What building it changed
1. Backprop is lineage tracking that carries blame
Built: micrograd, a scalar autograd engine, twice. Once following along, once from memory the same day to see what stuck.
Micrograd is about 100 lines. The core is a Value class where every value remembers what produced it: its parent values and the operation between them. If you have ever built column-level lineage for a data platform, you have already built this shape.
The difference is what flows through it. My lineage graphs answer “where did this bad row come from” once, during an incident, with a human reading the graph. Autograd answers the same question numerically, thousands of times per second, and then fixes the upstream automatically. The backward pass walks the DAG in reverse topological order and hands each node a number that means: this is how much you were responsible for the error.
def __mul__(self, other):<br>out = Value(self.data * other.data, (self, other), '*')<br>def _backward():<br>self.grad += other.data * out.grad<br>other.grad += self.data * out.grad<br>out._backward = _backward<br>return out<br>The += is the part worth staring at. When a node fans out to two consumers, blame arrives from both, and you sum it. Any data engineer who has debugged a fan-out DAG already has the intuition. It just runs the other way here.
Tip
If you have built data lineage, you own the mental model for autograd. Same DAG, opposite direction, and the payload is blame instead of provenance.
2. The loss function is a data quality metric
Built: bigram, trigram, and MLP language models on a dataset of 32,000 names.
Cross-entropy sounds like math-department vocabulary. From the inside it is brutally simple: the model is graded on how surprised it is by your data. The loss is the average negative log probability the model assigned to the character that actually came next.
And the number is interpretable. A loss of L means the model is effectively choosing between about e^L equally likely options at each step. The names dataset has 27 characters. Guessing uniformly gives ln(27) ≈ 3.30. Here is what each model bought me, numbers straight from my commit messages:
ModelLossEffective choices per characterUniform guessing3.3027Bigram (counting)2.45~11.6Trigram (neural)2.36~10.6MLP (dev split)2.19~8.9<br>Every drop in that table is structure the model found in the data. No structure, no drop. I sign off on data quality dashboards at work. Null rates, freshness, schema drift. Cross-entropy prices something none of them touch: how much learnable signal the dataset actually contains.
3. Batch size is a hyperparameter, and batch norm couples your rows
Built: MLP with activations, gradient diagnostics, and batch norm, following makemore part 3.
In my world, batch size is a throughput knob. Rows are independent. Processing 500 or 5,000 at a time changes cost and latency, never the answer. Row-level independence is something I have defended in design reviews.
Training violates both, on purpose.
First, the gradient is averaged over the batch, so batch size sets the noise level of every optimization step. Small batches give noisy gradients, large batches give smooth ones, and that noise changes what the model converges to. Batch size is not an efficiency setting. It is a hyperparameter that changes the result.
Second, batch norm. Each activation gets normalized using the mean and variance of the current batch. Your row’s forward pass depends on which other rows it happened to ship with. Rows in a batch see each other. From a data engineering purity standpoint this is horrifying. It also worked well enough to be load-bearing in deep nets for a decade.
The...