Introducing alp-data: A Shared Data Layer for Animal Language Processing – Earth Species Project

Introducing alp-data: A Shared Data Layer for Animal Language Processing

July 22, 2026

Image credit: Deborah / Adobe Stock

Key Takeaways

alp-data

is a pip-installable Python package that provides unified access to 35+ bioacoustic datasets through one consistent interface.

This removes the per-dataset engineering work that often stands between researchers and the large, diverse training corpora required to study animal communication at scale.

alp-data is the first piece of shared infrastructure for

Animal Language Processing

(ALP), an approach to studying animal communication that is AI-powered, data-driven, and species agnostic.

We are also introducing the

ALP Data Explorer<br>a visual tool to help anyone explore a subset of the cross-taxa data contained in alp-data.

Important note: We compiled and unified these datasets. We did not create them. Each one remains governed by the license its original creators set.

alp-data is free and hosted by ESP.

Turning Fragmented Data into Shared Infrastructure

The study of animal communication is moving toward large-scale, AI-driven inquiries across populations, species, and taxa. We introduced the term Animal Language Processing (ALP) as a name for this emerging field of study that aims to answer questions like: How do animal dialects emerge across populations or geographies? How does communication change across populations with unique dialects?

That shift – towards a data-driven, AI-powered, and species-agnostic approach – requires far more data infrastructure than the field currently has. While there’s been an explosion in bioacoustic and other ecological data, much of it remains out of reach for training and evaluating large AI models.

Today, we’re introducing alp-data, the data access layer we use within our own organization, and one of the first pieces of Animal Language Processing infrastructure.

Figure 1: Taxonomic diversity represented within the alp-data collection. These sunburst charts illustrate the breadth of mammalian (left) and avian (right) taxa currently included, with each ring representing progressively finer taxonomic levels from class to order and family.

As we trained and evaluated our models across increasingly diverse datasets, we found ourselves repeatedly solving the same data-access problems. Over time, those internal tools evolved into a unified access layer that became part of our own research workflow. Today, we’ve open-sourced it so that the wider field can work from the same foundation.

Access alp-data

The Animal Language Processing Bottleneck

The abundance of bioacoustic data is, by now, well established. Passive acoustic monitoring, animal-borne tags, and sustained recording efforts have produced data across species and environments at a scale that would have been difficult to imagine a decade ago.

Now, the challenge is in turning that data abundance into models capable of meaningful generalization.

Models generalize when they can perform well on data they have never seen before. In animal communication research, that means transferring what they have learned across species, taxa, and recording conditions. Achieving that kind of generalization requires training on diverse data at breadth and scale.

“The biggest lesson that can be read from<br>70 years of AI research<br>is that general methods that<br>leverage computation<br>are ultimately the<br>most effective,<br>and by a<br>large margin.”

– Rich Sutton,

The Bitter Lesson

Our own work on bioacoustic encoding pointed in this direction: models like NatureLM-audio that are exposed to a wide diversity of training data, including general audio alongside bioacoustic recordings, generalize more reliably across tasks and species than smaller, domain-specific models.

Yet building models that generalize across species requires substantial data engineering before any machine learning begins.

Imagine you’ve identified five datasets that could help answer your research question. The problem? They are hosted on multiple platforms. BirdSet uses one schema, Xeno-Canto another, Watkins yet another. Each comes with its own sampling rates, file structures, metadata conventions, and licensing terms. Before you’ve trained a single model or run a single experiment, you find yourself spending days or even weeks making these datasets work together.

Combining even a handful into a single training set requires writing and maintaining a bespoke data loader, alongside paying for storage, testing for consistency, tracking lineage and provenance, and allowing dataset schemas to evolve while maintaining backwards compatibility across versions.

Every additional taxon (let alone other modalities like video or behavioral notes) enlarges the surface area of formats to be handled, so that the corpora most valuable for cross-taxa work – the broadest and most diverse – are also the most difficult to construct.

What...