Structured Evaluation Pipelines to Improve Your AI Workflows | Philip Heltweg
Go back<br>Structured Evaluation Pipelines to Improve Your AI Workflows<br>21 Jul, 2026<br>Are you a startup founder applying AI agents or LLM workflows to a complicated domain problem? You already have a working prototype and now want to improve the quality of its output while making sure it doesn’t produce errors along the way?
With the sheer number of ways to manage AI context, tweak prompts, and swap agent harnesses, it’s hard to know what actually moves the needle. And when a new model releases, whether a stronger frontier model or a cheaper one, how would you quickly evaluate the impact on your product?
We recently worked with a startup on exactly these questions and I wanted to discuss our approach, rooted in AI engineering, at a high level. By the end you will have a working mental model and a concrete starting point for building this pipeline yourself.
The Overarching Idea
The core idea comes from traditional software engineering: you cannot improve what you cannot measure. So the first step is to define a way to measure the current quality of your system, set a baseline, and then build an automated pipeline to continuously evaluate against that baseline, without requiring new human intervention or relying on your personal impression of output quality.
Also from software engineering comes the idea of regression tests. In our case, we set up automated tests to catch safety-critical or otherwise dangerous regressions whenever you make changes or switch to a new model.
Offline vs. Online Evaluation
To evaluate the quality of AI responses, you can split approaches into two broad categories depending on what kind of inputs they use.
Offline evaluation means you build a static dataset of inputs you want to test, define a fixed evaluation suite, and measure quality against those test inputs. You can run local experiments, evaluate changes locally, and over time improve quality against your test data. This article focuses on offline evaluation as a practical starting point.
Online evaluation, by contrast, means evaluating actual production traces as they come in, flagging low-quality responses in real time or routing them for human review. It is more powerful but also more complex, and something you would layer on top of a working offline setup.
A Quick Note on Terms
Wording and concepts vary between different approaches to automated quality evaluation, so here is how I will use key terms throughout this article.
A trace is a single chat between a user and an LLM or AI agent, including its metadata. Multiple traces combined form a dataset , which is the core of offline evaluation. Traces can be annotated , either by humans or by AI agents, with scores against quality criteria. A collection of these annotations is your evaluation suite . Running an evaluation suite against a dataset to measure the effect of a local change is called an experiment .
graph TD<br>T["Trace (user/AI chat)"] -->|collected into| D[Dataset]<br>A["Annotation (quality rubric)"] -->|collected into| ES[Evaluation Suite]<br>ES -->|included in| E[Experiment]<br>D -->|included in| E<br>E -->|produces| R["Results (scores + report)"]<br>Offline Evaluation
During offline evaluation, you first generate a dataset of representative chats between your users and the LLM agent, ending with the last user message (because you are evaluating the AI’s response to it).
An easy way to build and extend these chats over time, keeping them human-readable to non-engineers, is to save them as Markdown files and load them in code. Here is an example trace:
id: account-cancellation-refund<br>description: Customer requesting a refund after cancellation outside the refund window<br>scenario-type: policy-edge-case<br>safety-response: false
# Assistant<br>Hi, I'm here to help with your account. What can I do for you today?
# User<br>I just cancelled my subscription but I didn't realize I'd lose access immediately.<br>I only used the service for two days this month. Can I get a refund?
# Assistant<br>I understand that's frustrating. Our policy covers refunds within 24 hours of a billing<br>cycle start. Since your billing cycle started 18 days ago, you wouldn't qualify under<br>the standard policy. Is there anything else I can help you with?
# User<br>That seems really unfair. I barely used it. I've been a customer for three years.<br>Surely there's something you can do?<br>And the corresponding Python to load it:
import glob<br>import re<br>from dataclasses import dataclass<br>from pathlib import Path<br>from typing import Literal
import yaml
Role = Literal["user", "assistant"]
@dataclass<br>class Turn:<br>role: Role<br>content: str
@dataclass<br>class Scenario:<br>id: str<br>description: str<br>safety_response: bool<br>turns: list[Turn]
def _parse_frontmatter(text: str) -> tuple[dict, str]:<br>if not text.startswith("---"):<br>return {}, text<br>end = text.index("---", 3)<br>return yaml.safe_load(text[3:end]), text[end + 3:]
def _parse_turns(body: str) ->...