MediumAudience, not audio: Representation learning from listener behavior | by WMG Lab | Aug, 2026 | WMG Innovation LabSitemapOpen in appSign up<br>Sign in

Medium Logo

Get app<br>Write

Search

Sign up<br>Sign in

WMG Innovation Lab

Stories from the front lines of music innovation. Warner Music Group’s engineering, product, design, infrastructure and data teams shipping at scale.

Machine Learning

Artificial Intelligence

Data Science

Music Technology

Recommendation System

Audience, not audio: Representation learning from listener behavior

By Paul Beata and Arvind Narayan

WMG Lab

5 min read·<br>1 day ago

Listen

Share

Catching a vibe<br>How do you define the “vibe” of a song with machine learning? Teaching machines to do this well is the backbone of modern audio discovery, enabling powerful search tools, powering automated content feeds, and organizing massive global catalogs. For several years, the answer has always begun with the same steps. You start with some hand-waviness about Mel-Frequency Cepstral Coefficients, and then you extract music tags. Perhaps, if you look at enough tracks, you start to get some interesting clusters. For example, if two tracks share the same BPM, valence, and instrumental profiles, they must be similar. But as anyone who’s experienced a playlist transitioning from a 2003 emo anthem to a modern pop-punk track can tell you, “sounding similar” is a small fraction of the story. How can we move past sonic similarity to gain some insight into which tracks share some “Cultural DNA?”<br>The objective of representation learning for music is to translate a song, a complex, emotional entity, into a structured mathematical framework. By transforming the intangible details of a song into vectors, we can treat each track as a unique coordinate in a broader latent space. With enough vectors, we end up with an expansive, multi-dimensional similarity map. In this digital universe, the distance between two track vectors represents their proximity. Similar tracks are clustered together, while different songs are far apart. Using this methodology, thinking deeply about what factors to include in the song vectors is critical. Without coordinates, after all, we’d have no map.<br>In this post, we’ll discuss how we moved beyond simple metadata and audio similarity to build a map of different musical neighborhoods. By leveraging Alternating Least Squares and collaborative filtering on hundreds of millions of listener-track interactions, we built a latent space where songs are grouped by the way people actually listen to them.<br>Why audio representation isn’t enough<br>While music AI traditionally begins with analyzing audio-based vector embeddings, our research demonstrated that they are misaligned with how music is actually consumed. In other words, songs that sound similar don’t necessarily stream similarly.<br>To prove this, we performed what we called a “cluster synchrony” evaluation. First, we used the Contrastive Learning of Musical Representations (CLMR) framework to generate 512-dimensional audio embeddings for tens of thousands of tracks in the WMG catalog. Next, we built a latent feature space and developed a methodology to measure the behavioral similarity of tracks within their neighborhoods.<br>Clustering: We projected the audio embeddings into a lower, 12-dimensional space using Uniform Manifold Approximation and Projection (UMAP) and clustered them with Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN).<br>Adding in streaming data: We collected daily streaming data for every track over 120 days.<br>The leave-one-out test: For every track, we calculated the total daily streams for the rest of its cluster.<br>Correlation calculation: We calculated the Pearson correlation between each track’s streaming curve and the aggregated curve of its cluster neighbors.<br>Final scoring: These scores were averaged to produce an average intra-cluster correlation metric.<br>This intra-cluster correlation metric rested at 0.11, indicating a very weak signal.<br>Get WMG Lab’s stories in your inbox

Join Medium for free to get updates from this writer.

Subscribe

Subscribe

Remember me for faster sign in

Further evaluation revealed more problems. Only 38% of the audio-based clusters were considered “usable” (achieving a correlation above 0.15), leaving the remaining 62% as effectively random noise. When we reran this analysis with a larger dataset, the fraction of usable clusters dropped even further to just 4%. In short, while these songs shared a similar musical structure, they were not consumed by the same audiences at the same time. They sounded alike, but we still lacked true cultural context.<br>Leveraging co-consumption for effective representation<br>We shifted our methodology to a more human-centric approach, co-consumption representations. Co-consumption defines a track’s identity through the community of listeners who engage with it. When thousands of listeners listen to “Helena” by My Chemical Romance...