We cut RAG costs 5x without losing quality

Scaling RAG: Chunking, Reranking, and Cost Optimization — Darko Trpevski

Scaling RAG: Chunking, Reranking, and Cost Optimization Ship RAG to production and watch it fail. Here's what works: semantic chunking, hybrid retrieval, reranking, and how to cut costs 5x. I've watched teams implement RAG systems, and here's what I see: they build something that works locally, ship it to production, and then it falls apart. The chatbot that could answer questions perfectly on a 10-document test set starts giving garbage responses on real data. Cost spirals because they're embedding everything multiple times. Latency blows out because they're reranking with a model that runs in 5 seconds per query.

The problem isn't RAG itself. It's that most teams treat RAG as "throw documents in a vector DB and ask questions." That's not RAG at scale. That's a prototype.

Real production RAG is about making a hundred decisions: How do you chunk documents without losing context? Which embedding model gives you accuracy without eating your budget? When do you rerank and when do you skip it? How do you handle the fact that your top-10 retrieval results are garbage 40% of the time?

This post covers what I've learned from running RAG systems that actually work.

The Chunking Problem

Your chunking strategy determines everything downstream. Get it wrong and no amount of reranking or retrieval magic fixes it. Most teams use fixed-size chunks (512 tokens, overlap 50 or 100). This is fine for learning. It's terrible for production.

Here's why: documents have structure. A legal contract has sections. A technical doc has code snippets and explanations. A research paper has abstract, methodology, results. Fixed-size chunking ignores all of that. You end up with chunks that split sentences mid-thought, chunks that miss context because they're too small, or chunks that duplicate content excessively.

The Approaches

1. Fixed-Size Chunking (Quick, Wrong)

python

from typing import List

def fixed_size_chunking(text: str, chunk_size: int = 512, overlap: int = 50) -> List[str]: """ Naive fixed-size chunking. Fast but loses document structure. """ chunks = [] step = chunk_size - overlap

for i in range(0, len(text), step): chunk = text[i : i + chunk_size] if len(chunk) > 100: # Skip tiny chunks chunks.append(chunk)

return chunks

# This will split sentences, lose context, and waste tokens text = "The capital of France is Paris. It's known for the Eiffel Tower..." chunks = fixed_size_chunking(text) # Result: ['The capital of France is Paris. It's known for the Eiffel ', 'Tower...'] # ^ Garbage. Sentence got split.

2. Semantic Chunking (Better, What You Actually Need)

python

from sentence_transformers import SentenceTransformer import numpy as np from typing import List, Tuple

def semantic_chunking( text: str, model_name: str = "all-MiniLM-L6-v2", similarity_threshold: float = 0.5, min_chunk_size: int = 100, ) -> List[str]: """ Split text at semantic boundaries.

Algorithm: 1. Split text into sentences 2. Compute embeddings for each sentence 3. Calculate cosine similarity between adjacent sentences 4. Start new chunk when similarity drops below threshold

This preserves meaning and document structure. """ model = SentenceTransformer(model_name)

# Split into sentences (in production, use nltk or spaCy) sentences = text.split(". ") sentences = [s.strip() + "." for s in sentences if s.strip()]

if len(sentences) = min_chunk_size: chunks.append(chunk_text) current_chunk = [sentences[i]] else: current_chunk.append(sentences[i])

# Don't forget the last chunk if current_chunk: chunk_text = " ".join(current_chunk) if len(chunk_text) >= min_chunk_size: chunks.append(chunk_text)

return chunks

# Usage text = """ The capital of France is Paris. It's known for the Eiffel Tower. The Eiffel Tower was built in 1889. It stands 330 meters tall. London is the capital of the UK. It has Big Ben and Westminster Abbey. """

chunks = semantic_chunking(text, similarity_threshold=0.4) # Result: 3 coherent chunks, no split sentences, preserves meaning

In production, I use a hybrid approach:

python

from typing import List, Dict, Any import re

def production_chunking( text: str, source: str = "unknown", max_chunk_size: int = 1000, min_chunk_size: int = 100, ) -> List[Dict[str, Any]]: """ Production RAG chunking.

Strategy: 1. Preserve document structure (sections, subsections) 2. Chunk semantically within sections 3. Add metadata for filtering and ranking 4. Small overlap to catch cross-boundary information """ chunks = [] chunk_id = 0

# Split by markdown headers first (preserve structure) sections = re.split(r'\n#{1,3} ', text)

for section_idx, section in enumerate(sections): lines = section.split('\n') current_chunk = [] current_size = 0

for line_idx, line in enumerate(lines): line_tokens = len(line.split())

# If adding this line exceeds max, save chunk and start new one if current_size + line_tokens > max_chunk_size and current_chunk: chunk_text =...

We cut RAG costs 5x without losing quality

Related Articles

(no title)

Scientists reverse brain aging, with a nasal spray

NHS apologises and admits Palantir have access to identifiable patient data

AI has torched the market for junior programmers

Al Vigier: Canada's AI strategy shouldn't include secret Palantir bills