Reducing the Latency of the InterPlanetary File System with Multi-Level DHTs and Probing

The InterPlanetary File System (IPFS) is an increasingly popular distributed storage network, connecting thousands of peers at a global-scale and supporting millions of requests daily. As it grows, IPFS is facing serious scalability challenges, with store latency escalating to the order of minutes. Previous studies on Distributed Hash Tables (DHT), including Kademlia upon which IPFS is based, have shown that as DHTs grow, the increasing peer churn causes their routing tables to become outdated, leading to either high latency due to dead links or high overhead for updating them. In this paper, we address these scalability issues by introducing an effective routing table maintenance mechanism, which exploits a multi-level DHT adapted for IPFS, namely, Hierarchical Kademlia, to mitigate the maintenance overhead. Using real peer activity traces from the IPFS network as a basis, we extensively explore the performance of IPFS, the costs involved and the effectiveness of our scheme via large-scale network simulations. Our findings suggest that our design offers scalability to IPFS, delivering stable request completion times with low maintenance costs: the overhead of our scheme represents less than 5% of the IPFS data plane traffic.