Breaking the Black Box: How AI Embeddings Cut Cloud Costs by 50% While Boosting Matching by 65% | by WMG Lab | Jul, 2026 | WMG Innovation LabSitemapOpen in appSign up<br>Sign in
Medium Logo
Get app<br>Write
Search
Sign up<br>Sign in
WMG Innovation Lab
The Wall: Why SQL Stored Procedures Couldn’t Scale<br>The Paradigm Shift: From SQL to AI Embeddings, From Always-On to On-Demand<br>The Matching Engine<br>The Results: Doing More with Less<br>The Lesson Learned
Stories from the front lines of music innovation. Warner Music Group’s engineering, product, design, infrastructure and data teams shipping at scale.
Breaking the Black Box: How AI Embeddings Cut Cloud Costs by 50% While Boosting Matching by 65%
By Paul Beata, Patrick Cosmo
WMG Lab
6 min read·<br>Jul 1, 2026
Listen
Share
Every month, streaming platforms like Spotify and social platforms like YouTube send the music publishing industry a stack of CSV files. Inside each one: metadata for millions of songs they’ve streamed but don’t know who to pay the publishing royalties to. They call it the “Black Box”. The same files land in every publisher’s inbox. Whoever identifies their own catalog in those rows fastest and most accurately gets paid; the rest is left on the table. At Warner Chappell Music, the publishing division of WMG, Global Match solves this.<br>Quick context on the rights side: a recording is a specific recorded version of a song; the underlying composition (the lyrics, melody, and structure) is a separate piece of intellectual property. Record labels administer rights to recordings; music publishers like Warner Chappell administer rights to the compositions behind them.<br>Global Match is a tool that identifies recordings that belong to our ‘rights’ catalog within these CSV files. For a long time, Global Match did this with brute-force SQL string-matching against our catalog. As the volume of global music and publishing data skyrocketed, our cloud bills spiked right alongside it.<br>So, we decided to re-architect the entire system. By shifting from heavy database processing to in-memory machine learning, we did what sounds like a contradiction: we slashed our cloud operational costs by over 50% while simultaneously boosting our matching output by 65%.<br>When you are dealing with high-volume data matching, scaling up usually means spending up. Here is exactly how we pulled off the opposite.<br>The Wall: Why SQL Stored Procedures Couldn’t Scale<br>To understand how we fixed it, you have to look at where we started. Our original matching infrastructure relied heavily on Snowflake stored procedures and sequential string-matching.<br>On paper, it made sense initially. Our music catalog data lived in the database, so we processed it in the database. But strict string-matching is notoriously rigid. If an artist’s name had a typo, a missing middle initial, or a non-Western character, the database missed it. To catch those edge cases, we had to write increasingly complex, computationally heavy SQL queries.<br>As a result, our database compute costs were rising, the system could only process files with up to 30K rows, and a single batch could take 12+ hours just to churn through. The user experience was equally cumbersome: upload a file to a region-specific cloud folder and then wait until tomorrow to see the results. We were essentially throwing expensive cloud compute at a problem that required a smarter algorithmic approach. And while we were waiting on those 12-hour jobs, the same monthly files were getting processed by every other publisher.<br>The Paradigm Shift: From SQL to AI Embeddings, From Always-On to On-Demand<br>We realized we needed a smarter matching approach, and a move to on-demand compute. This led us to Global Match 2.0, where we abandoned the heavy database-layer processing entirely in favor of text embeddings and vector spaces.<br>To maximize this efficiency, we rebuilt our underlying cloud architecture using a highly parallel, event-driven model. Instead of keeping massive servers provisioned and waiting, we built a system that scales compute dynamically up and down. When a massive batch file drops, a fleet of lightweight workers spin up (AWS Lambda functions), process the data in parallel at a rate of 20,000 rows per minute, and immediately shut down.<br>The parallel processing takes place in 2 phases:<br>1. The Preprocessor : A “preprocessor” AWS lambda function is triggered by the upload of a new file to S3 (uploaded by the user via our UI). This lambda effectively shards each file for parallel processing by enqueueing multiple separate work requests into an AWS SQS worker queue. The lambda doesn’t literally split the incoming file into separate files; instead it points each work request to a range of bytes (rows) in the original file that denote the boundaries of the shard assigned to the work request.<br>Get WMG Lab’s stories in your inbox
Join Medium for free to get updates from this writer.
Subscribe
Subscribe
Remember me for faster sign in
2. The Worker Fleet: The...