Your CTO is fleecing you (and AWS thanks you for it) - Justin Edwards
Skip to content
JUSTIN EDWARDS
Consultant. Rubyist.
Problem solver.
JE
Justin Edwards
// on this page
📅<br>Book time with me
Or send a message
in
𝕏<br>🦋
I'm<br>extremely inactive<br>on social media.
Calendly<br>or the contact form is faster.
A founder I was advising had just been through a round of layoffs. Then I looked at his AWS bill.
I told him, with no particular drama in my voice, that I could save him the equivalent of a professional salary by turning things off. Not rewriting anything. Not migrating anything. Just turning off things that were sitting there, idle, billing him every hour.
He almost fell out of his chair. Not in the good way.
This isn’t a hit piece on CTOs
I want to be careful here, because the headline sounds like one and it isn’t. I have spent twenty years around technical leaders. The vast majority of them are not bad people. They are not fleecing anyone on purpose.
What they are is nerds with toys. And nerds with toys, given no resistance, will play with the toys.
Full disclosure: I am one of these nerds. I love AWS. I love wiring up infrastructure. The neckbeard impulse to orchestrate petabytes in the sky lives in me too. The difference, I hope, is that I learned early not to take liberties with other people’s pocketbooks. When it is my own money on the line, the boring choice gets a lot more interesting.
How this actually happens
The pattern at almost every small SaaS I get pulled into looks the same. It isn’t a conspiracy. It isn’t an alliance. It’s two well-intentioned people speaking different languages.
The CEO, who is usually not technical, says things they picked up from Harvard Business School, from investors, or from a competitor’s marketing page. They want the product to be “best in class,” “highly available,” “enterprise-ready,” “ready for the next round.” These phrases sound responsible. They sound like the things a serious operator should be asking for. They are also, almost always, completely disconnected from what the business actually needs at its current size.
The CTO hears those phrases and translates them. “Best in class” becomes “I get to use the best-in-class services.” “Highly available” becomes “we need multi-region.” “Ready to scale” becomes “Kubernetes.” The translation isn’t dishonest. From inside the CTO’s head, those are the obvious right answers. They are also, conveniently, the toys the CTO has been wanting to play with for months. AWS ships hundreds of services. Elasticsearch indexes things. Kafka streams events. Redis caches things. Microservices let you have many small repos instead of one large one. Each is genuinely cool. Each is a real service that real engineers have built real careers on.
The CEO doesn’t have the technical literacy to push back. The CTO doesn’t have the business literacy to push back on themselves. Nobody in the room is doing math against the runway.
That’s the fleecing. Not malice. Two earnest people, no friction between them, and a lot of pretty toys on the shelf.
What it looked like in real life
The company I told you about at the top of this post had, when I came in, six AWS accounts. A root account, plus separate ones for production, staging, development, logging, and one whose purpose I never figured out. Each account had its own permissions, its own networking, its own resources, and its own confusing relationships to the others. Three of the accounts were paying for high-tier AWS support, separately. Same company, billed three times.
The servers were enormous. The database was oversized for a database that was barely being queried. Plenty of unused resources were just sitting in regions nobody remembered creating. Elasticsearch was running even though the search needs of the app could be served by a Postgres index. A Memcached server and a Redis server sat side by side, doing essentially the same job. The whole thing had been “built to scale.” Built to scale to what, nobody could quite say. The product had real customers, but the customer count was a small four-digit number and the AWS bill was somewhere around ten thousand a month. About half of that was waste.
Performance, by the way, was bad. Slow pages, slow queries. But the performance problem wasn’t in the infrastructure. The performance problem was in the application. There were N+1 queries everywhere, several obvious database indexes were missing, and a few queries that ran on every page load could have been cached at the application layer in twenty lines of code.
The CTO didn’t need a bigger server. The CTO needed to spend two days with the slow log and a SQL EXPLAIN.
I cut the bill roughly in half without rewriting a single line of application code. I...