How we run Gemini at scale across billions of posts
API
Use cases
Resources
PricingRequest a demoLogin
Get a demo<br>Log in<br>Try for free
Engineering blog<br>May 29, 2026
How we run Gemini at scale across billions of posts
Post author<br>Iván<br>Senior Data Engineer
Using LLMs with billions of inputs in a multi-cloud setup<br>At Modash we sit on top of a creator-discovery dataset that grows by millions of posts every day. A growing slice of that pipeline now runs through LLMs.
This massive volume of inference adds up on our cloud bills and our operational complexity. In this article you will learn how we actually run an LLM against billions of inputs without going broke .<br>Why We Use LLMs<br>Is the AI hype worth it? Do LLMs have any real use beyond being a 24/7 chatbot? We think so, and over the last year we’ve shipped several production pipelines where LLMs are visibly improving the data we deliver to our customers.<br>Several of those pipelines exist to extract structured meaning from messy, multilingual, multimodal social content. Historically these were patchworks of regex rules, keyword lists, and hand-coded extractors. They scaled in lines of code, not in coverage.<br>Take a caption like “This is not a sponsored post” or “I’m not being paid for this promotion”: it contains every keyword the rules were looking for, but means the exact opposite. The only way to handle it correctly is to actually understand the content.<br>Those false positives are unacceptable as they erode the perceived quality of a product customers are paying for .<br>LLMs reframe these as language and vision tasks instead of pattern-matching ones. The tradeoff is cost, throughput, and validation — which is what the rest of this article is about.<br>Our solution
Press enter or click to view image in full sizeOur upstream data lives in Iceberg tables on S3 . Each LLM use case has a corresponding Airflow DAG that triggers PySpark ETL’s that read our curated tables and extracts the rows that need inference.<br>The AWS Batch jobs generate the JSONL files (with the Gemini prompts) and stores them in different GCP buckets (one per region to leverage as much as possible compute capacity, more on that below), a pub/sub detects the event and send the JSONL file to a Gemini Enterprise Agent Platform (using the Batch API as it’s 50% cheaper).<br>Gemini Enterprise (formerly Vertex ) will read the model to be used from the path of the file and will store the output using the same partitioning strategy. From there, we run a periodic sync job that pulls those output JSONLs into S3 and lands them as Parquet.<br>Each input row is identified by a unique ID that is also present in the LLM output.<br>Finally, the data is ready to be used and our scheduled EMR jobs generate the data that we produce for our customers. Check this link if you want to lear more about how we optimize our EMR jobs.<br>What each Batch job does<br>From there, for each Parquet file, Airflow triggers out one AWS Batch job that, at a high level, prepares our raw platform data so Gemini can digest it:<br>Reads the necessary post data and handles heavy I/O tasks: (like downloading and encoding media) so it has to be parallel or the job’s resources will be infra-utilized.<br>Encapsulates the data into Gemini requests: Each post (or batch of posts) is packaged into a single, self-contained request payload along with its prompt instructions and structured output schemas.<br>Aggregates these request payloads and writes them into a large JSONL file that rotates when it hits ~900 MB.<br>JSONL structure :<br>"key": "XYZ",<br>"request": {<br>"contents": [<br>"parts": [<br>"text": "<br>POST DATA<br>],<br>"role": "user"<br>],<br>"generationConfig": {<br>"temperature": 0,<br>"maxOutputTokens": 8192,<br>"responseMimeType": "application/json",<br>"responseSchema": {<br>"type": "object",<br>"title": "SponsoredPostBatchResponse",<br>"properties": {<br>"results": {<br>"type": "array",<br>"title": "Results",<br>"description": "List of analysis results. Must contain exactly one result per input post.",<br>"items": {<br>"type": "object",<br>"title": "SponsorIdentification",<br>"properties": {<br>"PYDANTIC OBJECT SCHEMA"<br>},<br>},<br>},<br>"thinkingConfig": { "thinkingBudget": 0 },<br>"systemInstruction": {<br>"parts": [<br>"text": "OUR SPONSORED POST DETECTION PROMPT"<br>}, ...<br>Why JSONLs are big<br>The 900 MB number is deliberate — Gemini’s hard input cap is 1 GB. But packing requests into large JSONLs does not save us token money.<br>What large JSONLs do save:<br>Concurrent-batch-job quota . Gemini Enterprise enforces a cap on how many batch jobs we can have running per region simultaneously. A 1 KB file and a 900 MB file each consume one quota slot.<br>GCS object operations . Gemini Enterprise writes results back as files mirroring the input shape. Thousands of micro-JSONLs become thousands of micro-output-files, all of which cost way more on GCS at billion-scale.<br>Downstream simplicity . Fewer files for our ingestion job to enumerate, glob, and merge.<br>Spreading load across GCP regions<br>Gemini Enterprise AI Batch quotas are per-region , not per-project. If we...