How we built a video ingestion pipeline for 60× scale headroom | Codo

Home<br>Workshops<br>Blog<br>Open Source<br>Careers

Hire Us

← Back to blog

0.5 to 2M<br>Items a day, in production

~30M/day<br>Designed & load-tested for

~5 min<br>Publish to searchable

Mocks, across ~550 tests

Most systems are built for the load they have. This one had to be built for a load nobody could name yet. When we started, the target was “around 30 million items a day.” By the time we were in production it had been re-baselined to 2 million, with the 30M number still standing as the direction of travel. Somewhere in that range was the real system, and we had to build something that didn’t need rewriting when the number moved.

This is a note on how we did it: the architecture, the decisions that created the headroom, how we knew it worked, and three moments where the system pushed back. We were a small mixed team, a couple of platform consultants plus the client’s own engineers. Everything below is “we”; the hindsight at the end is mine.

The domain, in one line: ingest high-volume public video content from across the open web, enrich it (metadata, downloaded media, machine transcription), index it, and expose search over it.

What we were building

The job was a searchable, near-real-time index of public video : not a crawl-once dataset but a continuously updating one. As videos are published across the open web, the system finds each one within seconds, downloads the video and pulls its metadata, works out whether anyone is speaking and in what language, transcribes the speech to text, and makes the whole record (caption, transcript, metadata) searchable within about five minutes of the video going live. A user can then query “show me recent videos that mention X” and get answers off a corpus that’s minutes old.

Three forces pull against each other the entire time: volume (a firehose that could reach tens of millions of new items a day), freshness (minutes, not hours), and a scope that kept moving as the product found its shape. Almost every decision below is about holding those three in tension, so that when one of them changes (the volume target changed by more than an order of magnitude mid-project) the design bends instead of breaking. And the volume was a guess. We were building for a number nobody could pin down, which is a different problem from building for a big one.

t = 0<br>Published<br>a new video goes live on the open web

+ seconds<br>Discovered<br>we confirm a real, live video actually exists

Fetched<br>download the video to storage; look up its metadata

Language-detected<br>is there speech, and in which language?

Transcribed<br>speech becomes text, per language

≈ 5 min<br>Searchable<br>indexed and queryable through the API

Six stages, no shared state, queues in between

The pipeline is six stages joined by queues: discover → fetch → detect-language → transcribe → index → search. Each stage is its own service (containers on ECS/EC2) with its own input queue and its own autoscaling policy, and it knows nothing about its neighbours beyond a message schema. Services physically can’t import each other’s code; a build rule fails the compile if they try. That reads like a style rule; it’s the decision everything else depends on. Because the only coupling between stages is a queue, each one scales on its own backlog and a slow stage becomes a growing queue rather than an outage propagating upstream.

architecture · every service seam is a queue<br>OrchestratorassignmentsWorkersShared egressinfrastructureThe open webSNSdiscoveredFetcherObject storemediaMetadata APIlookupdownloadSNSMediaFetchedmetadataaudioAudio servicelanguage detectionTranscribersper languageSNSper-languageIndexer3 producers → 1 docdetectiontranscriptSearch indexwriteSearch APIreadSQS queue · buffer · DLQSNSSNS topic · fan-outdirect call · HTTP / storage

Every service-to-service hop is an SNS topic feeding an SQS queue (mint). That’s the decoupling: a slow consumer just grows a backlog, and each queue carries a dead-letter queue and its own autoscaling signal. The fetcher publishes one topic that fans out to two queues (indexer + audio). Direct HTTP/storage calls are dashed grey. Workers and fetcher share the same egress infrastructure out to the open web.<br>Stages talk through queues fed by fan-out topics, so a producer can add consumers without knowing they exist. Message schemas are versioned as discriminated unions, so two versions can coexist on a queue during a rolling deploy. Autoscaling is driven off queue depth , not CPU. The honest signal for a pipeline is how much work is waiting, and CPU-based scaling lags it badly.

We chose managed queues over a streaming log and containers-on-VMs over Kubernetes. The language call was a real bake-off: a colleague wrote the scanner in Go, I wrote it in Node. Go came out about 15% faster on the hot path. We shipped Node anyway. Our stack is Node, and a 15% edge on one service didn’t justify carrying a second language. Every one of these was a...