How Does an LLM Request and Response Cycle Work? A Full Walkthrough - QAInsights
Skip to content
Share via:
Copy Link
More
In this blog post, we will see how an LLM request and response cycle works, from the second you hit send to the moment the last word lands on your screen. I will not throw a wall of transformer math at you. Instead, we will follow one single prompt through every stage of the journey, so nothing feels abstract.
A single prompt sent to a large language model passes through multiple distinct stages before a response appears on screen. The process begins with the client app building a structured JSON request, which then clears an API gateway before the input text is broken into numeric tokens and assembled into the model’s context window.
The model then runs a forward pass through transformer layers to predict one token at a time, repeating this loop until an end-of-sequence signal is reached. Tokens stream back to the client as they are produced and are converted back into readable text incrementally, which explains the word-by-word appearance of responses.
I got curious about mapping this out properly while building iamspeed.dev, my browser based LLM benchmarking tool. Before I could measure things like tokens per second or time to first token, I had to understand exactly what happens between "you press Enter" and "words start appearing." Turns out that gap has a lot more steps than most people assume.
Table of Contents
Toggle
The example we will follow
Let’s keep one example running through the whole post. Say you type this into a chat app:
You: What’s the capital of India?
And a few seconds later, the model replies:
Assistant: The capital of India is New Delhi.
Simple enough on the surface. But that one exchange touches a client app, an API layer, a tokenizer, a neural network with billions of parameters, a sampling algorithm, and a streaming pipeline, all before those six words show up in your chat window. Let’s go step by step.
Step 1: You hit send
The moment you press Enter, your chat app does not just fire off the raw sentence. It builds a structured request, usually JSON, that looks something like this:
"model": "some-llm-model",<br>"messages": [<br>{ "role": "system", "content": "You are a helpful assistant." },<br>{ "role": "user", "content": "What's the capital of India?" }<br>],<br>"temperature": 0.7,<br>"max_tokens": 100,<br>"stream": true
Notice a few things here. The system prompt is bundled in even though you never typed it. Any earlier messages in the conversation get bundled in too, since the model has no memory of its own between calls. And stream: true is already set, which matters a lot later. This payload gets sent over HTTPS to an API endpoint.
Step 2: The API gateway
Your request does not go straight to a model. It first hits an API gateway that handles the boring but essential stuff:
Validating your API key or session token
Checking rate limits, so nobody floods the system
Routing the request to the right model version and the right compute cluster, often based on region for lower latency
Logging the request for billing and abuse monitoring
Think of this layer as the bouncer and the traffic cop combined. Only after it clears the request does anything resembling "AI" actually happen.
Step 3: Tokenization
Here is where things get interesting. The model does not read English. It reads tokens, which are chunks of text mapped to numbers. A tokenizer breaks your sentence apart, usually using a scheme like byte pair encoding.
Our example sentence, "What’s the capital of India?", might get tokenized into something like:
["What", "'s", " the", " capital", " of", " India", "?"]
Each of these chunks maps to a specific integer ID from the model’s vocabulary, something like [1780, 434, 262, 3139, 286, 4881, 30]. The exact split depends on the tokenizer the model uses, but the idea is always the same: text becomes numbers, because that is the only thing a neural network can actually compute on.
This step matters more than people realize. It is also why providers bill you per token instead of per word or per character. A short word can be one token, a rare word can be split into three or four.
Step 4: Building the context window
Your new tokens do not go in alone. The system stitches together:
The system prompt tokens
Any prior conversation history tokens
Your new message tokens
All of this gets packed into the model’s context window, which is just a fancy term for "how many tokens the model can look at in one shot." Each token also gets a positional encoding attached, so the model knows word order, since without it, "India of capital the" and "capital of India" would look identical to the network.
Step 5: The forward pass
Now the actual model runs. Your token sequence flows through dozens of transformer layers stacked on top of each other. Inside each layer, two things happen repeatedly:
Self-attention , where every...