Data Engineers Should Be Held To The Same Standards As Bakers<br>· Radically Ethical Data Consulting

We can't find the internet

Attempting to reconnect

Something went wrong!

Hang in there while we get back on track

Data Engineers Should Be Held To The Same Standards As Bakers

By Jordan Andersen

on 04/10/24

I shipped a product on time, under budget, that is going to be used by lots of people. No corners cut, no requirements dropped, no fudging the definition of done. It's a data pipeline that ingests images which will be used as input to a computer vision model, and it's going to power sustainability projections going 25 years into the future. Everything is approved, everyone is happy.

When you read that, was there any part of you that was impressed? Did you send a small "Congrats" my way in your head? Or maybe you felt kind of jealous that something that was planned actually came to fruition?

Maybe we should be asking ourselves why this isn't the norm everywhere.

It's all a bit silly

At the core of it, really, it's a little crazy to think or feel any of these things when you consider the context. Someone transferred me money on a regular basis while I used the skills I possess to make something they needed. I simply did what I was paid to do. That's it. There's nothing spectacular about it. That is, until we take a serious look at the state of the software and data industry. Imagine I was a baker telling the same story...

I baked a loaf of bread. It's a sourdough made from a starter that I tended to for a week until it was delicately bubbling. The manager and owner of the bakery inspected the loaf and approved its light, airy quality — they were happy with the texture, too. Something I made was deemed good enough to sell to someone who relied on the bread to be their breakfast. The thing I baked ticked the boxes of a tasty loaf of bread, as planned, and the recipe I used will be adopted by the bakery for the next 25 years. Would delivering baked goods on time be perceived as even a little bit remarkable?

Arguments can be made that building a data pipeline is more complex than baking a loaf of bread. But those arguments hold as much water as the sieve I would use to filter the flour for my sourdough starter. Like software engineering, there is an art and science to baking bread (mind you, that statement should be reversed since bread-making has been around for millenia, but I digress...). And from a guy with Coeliac Disease, I promise you there are a lot of<br>horrible<br>outcomes when baking. The benefit that writing code has over baking is that you can copy-and-paste someone else's solution and run it immediately to figure out if it's any good. Then you can modify, reduce, and enhance your starter code into an elegant and effective product. To do that with sourdough, you need to use a bunch of ingredients and wait a week to see if your thing rises into a fluffy creation of deliciousness or collapses into a brick-like mound of salt and starch. The iteration cycle of the latter is way more expensive, both in terms of time and materials. Now what if I was a hypothetical surgeon instead of a hypothetical baker telling a similar story? I'll spare you the gruesome details of what an unsuccessful abdominal surgery might look like.

So why are we so amazed when a tech project is actually delivered on time? One answer is market comparison: most tech projects are doomed before they start. But it's embarrassingly wrong to define the probability of success of a data pipeline or web app on previous outcomes in this way. Really,<br>why<br>are we so amazed when a product is shipped? I pose this as a rhetorical question rather than one of warm-feelings-and-daisies while we sit through Retros on What Went Well, What Could Have Gone Better, and What Was Out Of Our Control. Regardless of the number of "ceremonies", if a team doesn't ship a single thing, none of them should have been hired in the first place, yet running into teams in this situation is the norm. Instead of looking for catharsis in a post-mortem after a project fails, I subscribe to the idea that we need to raise our expectations of the quality of the things we generate. We spend an enormous amount time in meetings, typing on a computer, and pushing code to production. It's unacceptable to chalk up non-delivery to "we didn't get the scope right".

A pandemic of mediocrity

This isn't a problem that's unique to the tech industry. I used to coordinate programs for persons with special needs in the recreation branch of a local government. I was put on a committee to review safety standards across our operations while being told that it would "greatly enhance your career opportunities". I was nominated to lead a sub-committee of this committee (as you do) that was tasked with developing emergency response procedures and protocols to respond to someone having a seizure. (I just want to pause and note how fucked this was. While I<br>was<br>an accredited allied health...