Basically this exists to solve one of my biggest pain-points with dataframe libraries and big data. It s basically impossible to lazily download data on S3 with polars unless you re processing it strictly sequentially, and even then, it won t work with arrow and numpy data.This allows you to download a multi-gigabyte file on S3 into a Polars DataFrame in 100ms.# Demo alloc = demandmap.S3Alloc( ./cache.bin , # number of blocks capacity=512, # one megabyte block (per request chunk size) block_size=1048576 ) buf1 = alloc.get(S3_PATH) # Big file assert buf1.nbytes 400000000 # But this takes ~100ms df = pl.DataFrame([buf1]) It s one of those problems that manages to be both simple and difficult. I d wager 5% of devs know what a memory map is (higher on HN) and I d wager 1% of devs who know what mmap is know you can catch page faults in user-space, and yet another 1% of those devs know that it s possible to do userfaultfd in macOS with truly obscure mach_send_msg calls.I d really like to build a cross platform user faulting library covering Linux and Windows too, because nearly everyone who s touched a dataframe has had this exact problem.