What Happens Inside a RAG Pipeline

What Actually Happens Inside a RAG Pipeline (And Why “Banking” Is the Perfect Example)

Saurabh Dwivedy

SubscribeSign in

What Actually Happens Inside a RAG Pipeline (And Why “Banking” Is the Perfect Example) A plain-English walkthrough of embeddings, vector stores, HNSW, and why Elasticsearch and vector databases are secretly cousins - even if a few degrees removed :)

Saurabh Dwivedy Aug 03, 2026

LLMs are the RAGe and Retrieval-Augmented Generation (RAG) is fascinating (okay bad pun!). Those even remotely familiar with ‘AIspeak’ have heard about RAG. A good initial framing of what it is is as follows → “the AI searches your documents, then answers using what it finds.” This article is an exploration of a few key ideas pertaing to RAGs, Transformers, Attention, Indexes and a quick comparison with the realm of text-based searching via systems like ELASTICSEARCH, SOLR etc. Thanks for reading! Subscribe for free to receive new posts and support my work.

The article does require reasonable technical familiarity with database concepts (the regular RDBMSs and Document DB ones) are and pivots from there to Vector DBs which make RAG possible. The essential mental model for RAG systems is that they provide a layer of grounding to the otherwise global LLM responses that one would get from AI (LLM) directly stripped of additional context- what I like to call as ‘Ground Truths’. So, for example, if a user wants to ask some HR related questions to Claude or any AI - they can certainly issue a query and Claude will answer on the basis of the global data it has been trained on. But if the user wants answers from AI grounded in the context of their company’s HR policies then the user must provide additional context in the prompt. RAG is the technique that allows achieving this. I end by touching over a topic that is gaining traction lately - viz. with all the recent advances in RDBMSs supporting the storage of vector embeddings - like pgVector and so on…are pure play vector DBs like pinecone, weaviate losing ground? The rest of the article is about how such systems operate under the hood. Step 1: Chunking

Before any AI touches your documents, they get broken into smaller pieces — usually a paragraph or a fixed window of a few hundred tokens, sometimes with slight overlap so meaning doesn’t get broken mid-thought. This is crucial: search a whole 40-page document and a single relevant paragraph buried inside it gets diluted into irrelevance. Search a paragraph-sized chunk, and precision goes up dramatically. Step 2: Tokenization

Each chunk gets split into tokens — sub-word pieces, not quite full words. Critically, tokenization is purely mechanical. Take the same word ‘bank’ appearing in two different contexts where the meaning of the word changes. I am banking on you.

I prefer ICICI as my banking partner.

In these two sentences the word banking means different things and the LLM must be able to tell. Indeed this is the secret sauce of AI - but at the stage of tokenization both these words will tokenize identically. Step 3: Contextual Embeddings

This is where the magic alluded to earlier actually happens. And this is so critical that it needs a somewhat more detailed exposition than I have done heretofore as understanding this clearly is important. Any transformer — to be sure there are several architectures around — produces numerical representations for tokens (text), and those representations are what get shaped into embeddings (so called embeddings-vector - a really high dimenstional space in tech-speak, essentially amounting to a large set of numbers). Two lineages are of importance here. BERT-based models are encoder-only, trained to read a chunk of text in both directions at once, which makes them a natural fit for producing a single meaning-representation — from this lineage arose dedicated embedding models like Sentence-BERT, BGE etc. GPT-based models are decoder-only, trained to predict the next word while looking only left-to-right; these were built for generation, not for summarizing meaning. The tokens are processed together using a mechanism called attention , which evaluates each token's representation based on the words around it. So “banking” near “on you” ends up mathematically different from “banking” near “partner” — same string, but different embedding vector, because the model analyzes the context around the word. Step 4: One Vector Per Chunk

The individual token-level vectors get pooled into a single vector — commonly 1536 numbers if you’re using a popular OpenAI model (though the actual number varies by model: Cohere uses 1024, Google’s models use 768, some lightweight open-source models use as few as 384). This single vector is what represents “the meaning of this chunk,” and it’s what actually gets stored in the Vector DB - explained in the next section. Step 5: The Vector Store

Picture a simple table: Chunk ID Vector Text 1 0.01, -0.45, ... ”This is not the end of the...

What Happens Inside a RAG Pipeline

Related Articles

(no title)

Scientists reverse brain aging, with a nasal spray

AI has torched the market for junior programmers

Al Vigier: Canada's AI strategy shouldn't include secret Palantir bills

Google will expand age checks on Android worldwide till the end of the year