Cache invalidation and deploy verification: when a correct fix never applies | Innellis Research

&larr; Research Journal<br>A fix can be correct, verified, and still not be in effect

2026-07-30 &middot; 4 min read

I run a nightly pipeline that re-scores around 1,000 instruments.

One of the model personas uses company fundamentals. For crypto, it should<br>simply abstain, because there are no company fundamentals to evaluate.

But for SOL, it wasn't.

The provider was being called with only the ticker symbol, and SOL is also<br>the ticker of a NYSE-listed solar company. So the system was scoring a crypto<br>pair using a solar company's P/E, ROE, and other fundamentals.

I added an asset-class check and tested it against live data. Crypto<br>returned nothing, equities still returned the correct fundamentals, and the<br>tests passed. I deployed it and rebuilt everything.

Then the nightly pipeline ran and still wrote all seven persona results for<br>SOL, including the two that should have abstained.

At first this made no sense. The fix was correct. The test was correct. The<br>new code was deployed.

The cache was keeping itself alive

The scores are cached for 26 hours so the dashboard does not recompute the<br>full universe every time somebody opens it. The nightly job runs every 24<br>hours.

That small two-hour overlap was enough to break the whole assumption.

Each nightly run was reading the previous night's cache entry before it<br>expired. It then used that old result and wrote a new cache entry, which was<br>still there for the next run. So the stale result was basically keeping itself<br>alive.

I already had a cache version, but the comment said to bump it when the<br>response dictionary gains or loses a key. That is also how I had always used<br>it. In this case the structure did not change. Only the meaning of one result<br>changed, so I did not think about bumping the cache version.

Four explanations that looked reasonable

I also wasted time checking the wrong Redis database, reading an old TTL as<br>if the entry had just been written, and wondering whether the scheduler was<br>using a different code path.

Each explanation looked reasonable until I checked it properly.

What finally exposed the problem was looking at the actual production data<br>after the nightly run and asking a very simple question: did the output change<br>in the way this fix should have changed it?

That is the part I had not verified.

What I'd take from it

Testing a fix proves the code works. It does not prove the running system is<br>actually using the new behaviour.

Caches, stale workers, old images, feature flags, bind mounts and similar<br>things all sit in that gap. The check that closed it was comparing production<br>output against what the fix should have produced — not against the<br>absence of errors, and not against the test suite.

This is one entry from the Innellis research journal. The platform itself is an<br>evidence-based investment research system<br>— a committee of model personas that score instruments nightly, with every run<br>stored append-only so predictions can be checked against outcomes.<br>The track record is public, including the losses.