Optimistic Provide: How We Made IPFS Content Publishing 10x Faster - Blog | ProbeLab Analytics

Optimistic Provide: How We Made IPFS Content Publishing 10x Faster

March 20, 2026

Dr. Yiannis Psarras

Dr. Dennis Trautwein

Introduction

Publishing content in Distributed Hash Tables (DHTs) has traditionally been a slow operation, especially when: i) the network is large, and ii) the nodes participating in the network are churning frequently. The case is no different for IPFS’s Amino DHT, which meets both of these conditions. The ProbeLab team identified this problem years ago, through extensive measurements, and proposed an optimisation that was shown to improve performance, i.e., reduce the content publication time by over one order of magnitude while simultaneously reducing the network overhead by 40%.

The optimisation was named Optimistic Provide and it only recently shipped as a default in IPFS’s Kubo 0.39.0! The study was published in IEEE INFOCOM 2024, and we point readers to the publication for more but denser details.

In this blogpost we provide an overview of the technique proposed by our team, alongside basic technical details and, of course, results that prove the validity of our initial claims regarding its effectiveness.

A big thanks to the IPShipyard team for their support along the way and the final push to get this feature into production. According to the results, it was worth the effort.

TL;DR

The basic ideas behind Optimistic Provide are the following:

While walking the DHT, immediately store records with peers that are likely among the 20 network-wide closest peers.

Terminate the DHT walk immediately when the set of the discovered 20 closest peers likely constitute the network-wide closest peers.

Return control back to the user after most (not all) of the PUT RPCs have succeeded and continue with the remaining ones in the background.

Points 1. and 2. require knowledge of the total network size which we derive from a lightweight estimation method piggy-backing on the routing table refresh mechanism to not incur any additional networking overhead.

Optimistic Provide decreases significantly the upload latency, from more than 13 seconds (often close to 20 seconds) to less than 1 second, and brings tremendous value and impact to the performance of IPFS.

What this means in practice: content publishers can now have their content published almost in real-time, i.e., ~1 second after content has been pushed to the network. This is a major improvement, compared to > 10 seconds before, as developers, users and application providers can re-iterate and debug in real-time.

Traditional Provide Operation

To understand how these optimizations work, we must first look at the traditional "provide" operation. In a Kademlia-based Distributed Hash Table (DHT), such as the Amino network used by IPFS, storing a record requires identifying the k peers closest to that data's identifier where k is set to 20 in the Amino DHT. In this context, "closeness" is not defined by geographical proximity but by the XOR distance metric. This metric calculates the distance between a peer’s unique PeerID and the data’s Content Identifier (CID) by performing a bitwise XOR operation on their IDs.

Starting from a hard-coded set of bootstrap peers, a node fills its local routing table. The process of finding these 20 closest peers is known as a DHT Walk . It is an iterative search where a node queries its local routing table for the closest known peers and asks them for even closer candidates. This cycle continues until the initiator has received successful responses from the three closest peers it has discovered. Once the walk terminates, the Follow-Up phase begins: the initiator pushes the provider record to all 20 of those closest peers to ensure data availability even if some nodes leave the network.

So the takeaway is: a provide process consists of two phases

DHT Walk Finding the network-wide 20 closest peers

Follow-Up Pushing the record to these 20 peers

Performance Bottleneck

While this system is robust, it is historically slow, often taking dozens of seconds or even minutes to complete. Our research at ProbeLab identified that the primary delay is the termination condition of the DHT walk.

The "traditional" algorithm is rigid: during the DHT Walk phase it insists on waiting for responses from the three discovered closest peers. In a permissionless network where peers frequently churn, these specific peers are often unreachable. The system then "backtracks," querying more distant peers to fill the gap, while the actual 20 closest peers might have already been discovered.

The graph below shows the cumulative distribution function of the total duration of provide operations from Europe, the traditionally fastest region. The original graph can be found here. The graph shows that the median latency is around ~20s and that in the worst cases a single provide operation can take over two minutes.

This...