How To Not Die By A Thousand Cuts. Or, How To Think About Software Quality.

WIP public β &raquo Project

"Writing for nerds".

Email:

|| Get RSS feed.

Contents

First off, what even is Quality?

All things emerge, change, and die. I think Quality is the experience of the process. The idea of Good Quality essentially boils down to performing the process with grace, and leaving the place better than we found it.

Further, the process of emergence and change—i.e. living—is also the process of dying. It follows that to think clearly about the Quality of the former one must think clearly about the Quality of the latter. The saddest way it can unfold is a slow painful degradation without healing succour meaning or hope. The proverbial death by a thousand cuts. I hope you never witness such a passing, even from afar.

Ok, that got dark fast, and if we're not careful, we will produce a 300 page Zen dialogue on Motorcycle Maintenance. So we will distract ourselves with the much smaller, lighter—and I'd argue, even pleasant—task of contemplating Quality of software products.

None of what follows is novel, but I feel the message and its surrounding context bears repeating, because if it is not obvious already, software fails us all the time. Far too often with terrible consequences.

What is the nature of software products?

See? This is already easier than asking "What is the nature of life?".

Like any other machine, a software product is wrought of the labour of many minds and hands, and it requires maintenance and upkeep throughout its life.

Unlike all other machines, it is pure concept, and as such it is infinitely malleable and mutable. And mutate it does, all the time.

Sometimes, "finished" software emerges, only needing minor fixes and patches, but remaining the same in purpose, interface, and behaviour. Many Unix tools fall in this category. Some projects like ZeroMQ make it their explicit goal. Many Clojure programmers value such "finished-ness". Such scattered examples exist.

Most software does not have this luxury. Most software must change indefinitely because the world it must serve changes indefinitely. The Emacs editor is a software product that has evolved non-stop for nearly half a century since it emerged in 1976, and it continues to thrive. This post was written in Emacs.

There is a strong reinforcing feedback loop too. Software changes the world fast, forcing software to change faster. The current reincarnation of Machine Learning and AI can be viewed as an expression of this process. We're basically saying it's all accelerating so much that it is getting humanly impossible to write and revise software fast enough, to out-OODA the pace of change. So we must instead find algorithms that sense the world and then dynamically generate or revise other algorithms to achieve system objectives (viz. alter the world further in our favour).

We have to wonder, how do we make sure our product continues to thrive and succeed under such unrelenting pressure of constant and sometimes violent change? And who's neck should be on the line for it?

Whom to hold responsible for Software Quality Assurance?

The Usual Suspects?

Those "Quality Assurance" boffins? Developers? UX people? DevOps?

The Less Usual Suspects?

Product managers? Analysts? Customer success? Sales? Marketing?

The CEO?

The AI?

Consider the scenarios below. All of them directly impact customers, making them think "bad quality". Consider who is responsible for the underlying problem (or more likely, problems)?

Your app framework is extremely performant and glitch-free. Your app bombs.

A feature does exactly what it promises, but people fail to use it right.

Your company committed half of itself to ship a second product in record time, but customers never really wanted it.

A huge update was pushed out on a do-or-die basis. Naturally it misbehaves, can't be rolled back, costs 5x as much to get right as it took to ship, and the rework effectively adds months to your plan of world domination.

Your service fails to scale. You discover there were no benchmarks.

A deployment breaks production. You discover a bad configuration.

A feature leaks data to unintended users and breaks SLAs / regulations. Your CEO releases a statement blaming a DevOps engineer.

A several-hour glitch goes un-monitored, causing serious widespread data corruption.

Your production noticeably degrades often. A large sea mammal is your mascot.

Your production seldom degrades, but when it does, it takes down half the known Internet along with it.

and on and on…

In a quiet moment of honest self-reflection, you may confess to the mirror that the thousand cuts metaphor applies. That any of the above scenarios were likely the product of corner-cuts, often near-invisible to the naked eye in the moment. Corner-cuts that added up—nay, compounded—over time; slowly as band-aids, then as stitches and casts, and then suddenly as gangrene. And maybe the whole thing died of...