Claim Checks at Scale: Stop Letting Every Consumer Talk to S3 // sprawl.sys
claim-checks-at-scale-stop-letting-every-consumer-talk-to-s3.md<br>Claim Checks at Scale: Stop Letting Every Consumer Talk to S3<br>Ideas on implementing claim checks with events at scale by leveraging a gateway for the read side$ logged August 7, 2026<br>#enterprise-integration-patterns<br>#claim-check<br>#event-driven<br>#architecture<br>#messaging<br>#microservices<br>#api-gateway<br>#aws<br>#caching
Previously during the Advent of EIP I covered the Claim Check Pattern as part of Day 5: Message Types & Event Payload Strategies. In that write up I left the implementation purposefully simple: write the document to S3 and include the S3 address in the payload for the consuming side to receive and fetch the bytes. This is a pattern I have used time and time again and it works well regardless of whatever object store you use and is very simple for your team to reason about.
Recently I worked with a team that wound up having an interesting problem… multiple downstream services needed to fetch the payload specified by an event type. The simple solution was to wrap the details of fetching the payload in a library to encapsulate the bucket layout structure and replicate the IAM permissions to read from the bucket to each consuming service.
The Gap: Tight Coupling with Object Storage<br>It worked, but I didn’t really like it. It was really bugging me for a few reasons:<br>IAM Sprawl - any new service utilizing this library would also need to replicate IAM permissions for reading the bucket. Implementation becomes a library addition + an IAM permission add on<br>Migration Coordination - any changes to the object structure would need to be rolled out in coordination with consuming services, some owned by different teams. Achievable, but felt like it would be pretty heavy-handed!<br>Encapsulation at the Wrong Layer - Wrapping the access patterns was better than nothing, but felt like the same smell I have seen in my career where many services share the same database via a library of ORMs. Well intentioned, but messy. Long ago we solved that problem by having services access shared data via an API to provide better abstraction over shared libraries and this felt like repeating that old mistake.<br>While this all sounds great technically, this tight coupling has a tangible end-user cost… evolution of architecture to fit new services and features come at a heavy cost due to the amount of coordination and mitigation of impacts involved. I’ve seen many projects like this over my career and they tend to draw out way longer than they need to.<br>Solution: Encapsulate the Document Store with a Gateway<br>I’ve already touched on what the solution here is while talking about the database analogy: there is a preference to wrap a database in an API rather than letting multiple services hit it directly. The object store for claim-checked documents is a database, so we should give it the same treatment. For the write side I won’t talk about that today, we can assume that the application writing the claim check owns the data and as a result is allowed to have intimate access to the storage. Let’s focus on the read side instead.<br>The first approach one might go after on this is to simply wrap the object store in an API, and that isn’t a terrible idea, it is perfectly fine for most use cases. However the appetite to build and own a new service can be low for some teams, so why not rely on something purposefully built for this scenario? nginx-s3-gateway can be a great fit here. It’s already a battle-tested solution and provides some other interesting features like caching for frequently retrieved objects.
The architecture at this point becomes effectively an API that consumers talk directly to using a common identifier that the publisher provides as the claim check key. The power of using nginx-s3-gateway becomes more relevant when you factor in the caching layer it provides, since if six consumers retrieve the same object, only the first request hits S3 and the rest are served from cache.<br>The caching is layered in memory as part of nginx, so if you are running a fleet of multiple instances you will want to keep that in mind. You’ll still benefit from some caching, but not every identical request will hit the cache. If you DO need a shared cache, you might look into some other options such as OpenResty with Redis cache access.<br>Bonus: The Route Table as a Migration Lever<br>There are a lot more configuration options you can put in place here, even using something like Envoy to control the route and then completely change the backing store to something besides straight S3 reads if you desire! Once something like Envoy is sitting in front of the read path, the route table stops being just a security boundary and becomes the place where you decide how a claim check gets resolved, not just whether it’s allowed. You could match on the path, a header, or...