Caching Is Not Free | Panayiotis Kritiotis

Cache is one of the most misunderstood components in software architecture.<br>We all know this story.<br>You have a critical web application and you notice that the latencies of a few endpoints have gone up in the last few weeks.<br>You sit down to check what’s happening and you find out that there’s actually nothing wrong. The load has increased organically over time and it now threatens your latency SLAs.<br>So you schedule a meeting, and you get into a room with your team to see what you need to do.<br>What&rsquo;s the first instinct?<br>Easy! Let’s throw a cache in front. Add Redis in front of your service and all of your problems are solved.<br>Well, not quite!<br>Unfortunately, many times the instinct becomes a decision.<br>It’s not free performance - it’s a tradeoff<br>We often think of cache as free performance when it’s really just another fundamental design decision.<br>And as every design decision in distributed systems - guess what - it’s a tradeoff!<br>When you choose to use a cache, you’re buying low latency but you’re paying with data freshness/staleness and complexity.<br>And we’re not talking about trivial complexity. Remember the famous cliché of the two hard things in software engineering?<br>Cache invalidation is just hard.<br>And on top you now have synchronization edge cases to solve and most important of all: a whole new infrastructure to maintain.<br>Where it bites<br>The thing with cache is that it’s very easy to include it in your architecture, and this often introduces side effects that in the long-term can harm your system instead of helping it.<br>The Masking Problem<br>Many times we unconsciously use the cache as a bandaid fix when the database performance degrades. Been there, done that.<br>The reality though is that many times the problem stems from poor engineering practices.<br>I have seen many cases with N+1 queries, missing indices or simply badly written code that could be optimized simply and clearly and would solve the latency issue without further action.<br>Note here that the issue I&rsquo;m focusing on is not the cover-up of the real problem per se. It&rsquo;s the ignorance.<br>As long as it&rsquo;s a conscious decision, the bandaid can be the correct solution. Time-pressure or the high fix complexity can make a good case. Remember, it&rsquo;s all tradeoffs.<br>If it&rsquo;s just another tech debt decision, all good! Think about it, document it, and tackle it later.<br>By adding a cache to cover up these inefficiencies without a proper analysis, you’re not fixing the root cause - you&rsquo;re just masking it.<br>It&rsquo;s not only a waste of resources and an increase in maintenance and complexity - these issues stack up and given enough time and evolution of the service, these will come back and knock on your door - hopefully not at 2am.<br>High cardinality data & the read-heavy thinking trap<br>We often think that cache pretty much works for all read flows.<br>But if your service has highly variable data without a hot path and the requests are not similar, then caching just makes things worse.<br>Hit rates drop, you&rsquo;re doing evictions constantly and you’re in an infinite miss → load → evict cycle which ends up increasing latency and utilizing more resources.<br>High-cardinality data is one way of dropping your hit rates, but in the past I&rsquo;ve walked into another case.<br>A long, long time ago, I was assigned to design a user preferences service that had a non-negotiable: return the requested user preferences fast.<br>The service was simple. A user logged into the app and if they had already stored preferences, they would show up to modify their experience. If they didn&rsquo;t, users could save their preferences and these would be available in their next interactions.<br>How did I ensure that my service had fast reads? Cache of course!<br>I created the backend service that stored the data in a SQL database and threw a Redis in front to return the data blazingly fast. I even confirmed the promise of caching. Retrieval from the main SQL storage required ~100ms while Redis returned it in 2ms. 2!<br>Guess what! The hit/miss ratio was ridiculous.<br>Why? A re-read wasn&rsquo;t needed in the normal flow. A read was only needed on log-in and after storing new preferences. If I had to estimate, the ratio of read/write would be ~10000:1.<br>Interestingly the project was a huge success but this was not because of the caching. It was successful despite adding the cache.<br>The takeaway? Even if your app is read-heavy, and even if you don&rsquo;t have high-cardinality data, if your access patterns don&rsquo;t have hot paths, a cache is useless.<br>When a latency cache quietly becomes a capacity cache<br>One important distinction we often neglect is that not all caching is the same. Caches come in different types1. The main ones are:<br>Latency - boosts the latency of your application<br>Capacity - increases your load capacity. Without it you can&rsquo;t support an increased load.<br>There&rsquo;s a framing I really like from The Coder Cafe2 about the...