Making jsDelivr’s Origin More Reliable at Scale

Skip to content

jsdelivr.com

When the original storage index was designed, jsDelivr tracked roughly 350,000 unique files. By the time we replaced it, that number had grown to more than 600 million. We moved the index from Redis to MariaDB, migrated storage from Amazon S3 to Wasabi, compressed and cleaned up the stored data, and reduced the GitHub API work needed for uncached requests.<br>Most jsDelivr requests are served directly from CDN edge caches. The origin handles the less visible path behind them: resolving a package and version, retrieving or generating a file that has not been cached yet, storing the result, and returning it to the CDN.<br>This path matters most for newly published versions, less frequently requested files, GitHub branches and tags, and dynamically generated resources such as /+esm bundles. An origin problem can therefore affect cold or unusual requests while popular files continue to be served normally from cache.<br>Over the past few months, we completed two major groups of changes. The first rebuilt the metadata and storage layer used for cached files. The second reduced our dependence on GitHub API capacity when resolving GitHub-backed URLs.<br>A companion post covers the package-level /+esm improvements. This one is about the origin infrastructure underneath them.<br>From 350,000 to more than 600 million files<br>The original storage index was designed when jsDelivr had around 350,000 unique files.<br>The design anticipated substantial growth and was expected to remain practical at tens of millions of files – about one hundred times the initial scale. For a long time, it did.<br>The object-storage bucket itself was the source of truth. Redis kept a derived index for fast origin lookups. Once per day, the origin listed every stored file and rebuilt the complete index, while incremental updates kept it current between rebuilds.<br>That made consistency straightforward. If Redis lost data or missed an update, the bucket still represented the actual state, and the next complete listing reconstructed the index. At the scale we had then, both the listing load and the Redis memory requirements were reasonable.<br>By the time the index passed 50 million files, a full listing already took several hours. Other work took priority and the migration was postponed. When we returned to finish it, the bucket contained more than 600 million files.<br>Each rebuild now meant scanning hundreds of millions of files that had not changed, consuming substantial object-storage bandwidth, and keeping the complete index in memory on the origin side.<br>Replacing the in-memory index with MariaDB<br>The new implementation stores file metadata in one central MariaDB database.<br>For package files, records are keyed by the package source – npm or GitHub – together with the package or repository name, file path, and version. Additional fields identify generated variants such as minified or ESM files and keep the upload metadata needed to locate the physical file.<br>Normal origin requests use direct indexed queries. MariaDB can keep hundreds of millions of rows primarily on disk instead of requiring the entire working index to remain in memory.<br>We initially considered using a central database for writes and a local read replica on every origin. That would provide slightly lower read latency and keep local lookups available during a central outage, but it would also introduce replication lag, more components to operate, and periods when different origins could see different states. We ultimately chose a single central database and handle availability through the fallback path described below.<br>Keeping storage and metadata consistent<br>With MariaDB, the index becomes the source of truth for which files the origin considers available in storage.<br>The upload path writes the file first and creates the database record only after the upload succeeds. If a row exists, the corresponding file was successfully stored.<br>The reverse case is acceptable. A file upload may succeed and the following database insert may fail. Without the database row, the origin treats the file as uncached. The next request retrieves or generates it again and performs another upload.<br>The upload process attempts to remove the unindexed file after a database failure. If that cleanup also fails, the result is an orphaned file using some additional storage – not an index entry pointing to content that was never uploaded.<br>Database primary keys also prevent concurrent origins from indexing several copies of the same logical package file and version. If another origin completes the same write first, the later duplicate can be removed.<br>Falling back when the database is unavailable<br>If the origin cannot query the metadata database, it skips the object-storage lookup and fetches a fresh copy from the original npm or GitHub source.<br>The request may take longer and may repeat work for a file that is already stored, but the content can still be returned....