Dear Google Cloud: Your Deprecation Policy is Killing You | by Steve Yegge | MediumSitemapOpen in appSign up<br>Sign in

Medium Logo

Get app<br>Write

Search

Sign up<br>Sign in

Dear Google Cloud: Your Deprecation Policy is Killing You

Steve Yegge

23 min read·<br>Aug 15, 2020

74

Listen

Share

God dammit, I didn’t want to blog again. I have so much stuff to do. Blogging takes time and energy and creativity that I could be putting to good use: my novels, my music, my game, and so on. But you get me riled enough, and I have to blog.<br>Let’s get this over with, then.<br>I’ll begin with a small but enlightening story from my early days at Google. For the record, I know I’ve said some perhaps unkind things about Google lately, because it’s frustrating when your corporate alma mater makes incompetent business decisions on the regular. But Google’s internal infrastructure is truly extraordinary, and you could argue that there is still none better today. The people who built Google were far better engineers than I will ever be, as this anecdote should serve to illustrate.<br>First a wee bit of background: Google has a storage technology called Bigtable. Bigtable was a remarkable technical achievement, being one of the first (if not the first) “infinitely scalable” key-value stores: the beginning of NoSQL, basically. These days Bigtable still holds up well in the rather crowded space of K/V stores, but back in the day (2005) it was breathtakingly cool.<br>One fun bit of trivia about Bigtable is that they had these internal control-plane entities (as part of the implementation) called tablet servers, which had large indexes, and at some point they became a scaling bottleneck. So the Bigtable engineers scratched their heads over how to make it scale, and realized that they could replace the tablet servers with Bigtables to unblock the scaling. So Bigtable is part of its own implementation. It’s Bigtables all the way down.<br>Another cool bit of trivia is that for a time, Bigtables became popular and ubiquitous inside Google and everyone and their dog had one. So at one Friday’s TGIF all-hands, Larry Page casually asked in passing, “Why do we have more than one Bigtable? Why isn’t there just one?” Because in theory, one Bigtable should have sufficed for all Google’s storage needs. Of course they never did migrate to just one, for practical software engineering reasons (e.g. blast radius), but the theory was interesting. One database for the whole universe. (Side note: Anyone know if Sable does this at Amazon?)<br>Anyway, here’s my story, to get us started on my rant.<br>One day, after I’d been working at Google for just over 2 years, I got an email from the Bigtable engineering team. It said something along the lines of:<br>Dear Steve,<br>Greetings from the Bigtable team. We wanted to let you know that you are running a very, very old Bigtable binary in the [some data center name] data center. That version is no longer supported, and we would like to work with you to help you upgrade to the latest version.<br>Please let us know if you can schedule some time to work with us on this.<br>Best,<br>Bigtable Team<br>You get a lot of email at Google, as you can imagine, and when I glanced at this one, this is what I first perceived it to be saying:<br>Dear RECIPIENT ,<br>Greetings from Some Team. We wanted to let you know blah blah blah blah blah blah blah. Blah blah blah blah blah blah blah BLAH, and blah blah blah blah immediately.<br>Please let us know if you can schedule some of your precious time to blah blah blah.<br>Best,<br>Some Team<br>I almost deleted it on the spot, but there was this lingering, nagging feeling that it didn’t quite feel like a form letter, even though it obviously wasn’t for me, since I didn’t have a Bigtable.<br>But it was weird.<br>For the rest of the day, as I was alternating between working and deciding which species of gummy shark to try next in the micro-kitchens, of which there were at least three close enough to hit from my seat with a well-aimed biscuit, I thought about that email with a growing sense of mild anxiety.<br>They had used my name specifically. And the email had been sent to my email address and nobody else’s, and not by cc: or bcc:. The tone was very personal and pointed. Maybe it was some sort of mistake?<br>Curiosity finally got the better of me, and I went to look at my Borg console in the data center they’d mentioned in the email.<br>And sure enough, I was running a Bigtable there. Whaaaaat? I looked at its contents, and lo! It was from the Bigtable codelab I’d run back in my first week as a Noogler, in June 2005. The codelab had you fire up a Bigtable so you could programmatically write some values to it, and I had apparently never shut it down afterwards. It was still running there, over 2 years later.<br>There are several remarkable aspects to this story. One is that running a Bigtable was so inconsequential to Google’s scale that it took 2 years before anyone even noticed it, and even then, only because the version was old. As a point of...