smevals - a small eval suite for evaluating models, prompts, and harnesses | Prime Radiant

← Back to Blog

Jul 31, 2026

smevals - a small eval suite for evaluating models, prompts, and harnesses

We've been building a new system for running evals against different models, prompts, and harnesses, with the goal of being able to identify the most appropriate small and inexpensive models for different categories of task.

Frontier models continue to improve at an impressive rate, but those improvements are often accompanied by increases in price as well. GPT-5.5 and 5.6 Sol are twice the price of GPT-5.4. Claude Fable 5 is twice the price of Claude Opus 4.8. Even Google's inexpensive Gemini 3.5 Flash-Lite model has increased in price from Gemini 3.1 Flash-Lite.

Meanwhile the options for inexpensive models have never been more abundant. Local models that run on small devices are exploding in capability, and the leading edge open weight models are improving at a dramatic rate, and at a price point significantly lower than their proprietary competitors.

The smevals vocabulary: evals, tasks, configs, runs, and grades

smevals (for small evals) is a Python CLI tool that executes eval suites that are defined as a directory containing YAML configuration and executable scripts. Here are the key concepts of the smevals system:

An eval is a collection of challenges designed to answer a question about a model, for example, how good is that model at generating SVGs?

Each eval is a collection of tasks . A task is a specific challenge, for example "Generate an SVG of a pelican riding a bicycle".

When you run the eval you do so against one or more configs . Each config specifies a model to be evaluated, but may also include other parameters to test, such as different system prompts, model parameters, or agent harnesses.

A run records what happened when a specific config was used to execute a specific task. A runner is the script that executes a run.

Once you have collected one or more runs, you need to evaluate the results to see how well the model (or config) did. This is done by a grader , which produces a grade .

Each grader runs a sequence of checks . These can be simple operations, like checking for a specific string in the output, or confirming that the output is valid XML. They can also be more complicated custom operations (implemented as scripts called checkers ), including using other models to answer questions about the run.

The smevals process

To run an eval using smevals you need to:

Design and implement the tasks for the eval

Decide how you will be grading the outputs of that eval

Run the tasks against one or more model configurations

Run the grader to determine scores for those runs

The results of an eval can be examined either in the terminal or using a web application. The web reports can be baked out and published as static files.

Writing evals with a coding agent

The smevals README is designed for both humans and agents. A coding agent that reads that document should have everything it needs to know in order to construct an initial eval.

That README is also bundled with the tool, and is available using the smevals docs command.

This means you can start a session in Claude Code, or OpenAI Codex, or Pi, or your agent of choice, and prompt the following:

Run the command "uvx smevals docs"

Then, once the agent has read the resulting documentation:

Now build an eval that tests how well models can write haikus, with two tasks - a haiku about a pelican and a haiku about two otters in love

This should construct the eval in your current directory. You can then run it with this command:

uvx smevals run . -g

The . tells it to run the eval in the current directory. Adding -g causes it to grade the runs as soon as they have executed - without that you would need to run a separate smevals grade . command later on.

Here's the output I got from running this for the first time:

otters-in-love / default / gpt-4.1-mini ... ok (5.0s) -> runs/otters-in-love/default/gpt-4.1-mini/2026-07-24T01-07-45Z

grade: pass score=1.0

pelican / default / gpt-4.1-mini ... ok (12.6s) -> runs/pelican/default/gpt-4.1-mini/2026-07-24T01-07-50Z

grade: pass score=1.0

I ran it against two more models like this:

uvx smevals run . -g -m gpt-5.5 -m gpt-5.4-nano

That created four more runs, trying each of the two tasks against those two new models.

The runner used here is a short shell script called run-llm that calls the llm CLI with the model and prompt supplied by smevals, then saves the corresponding LLM JSON log as an artifact alongside the output:

#!/usr/bin/env bash

set -euo pipefail

llm -m "$SMEVALS_MODEL" "$SMEVALS_PROMPT"<br>llm logs -c --json > log.json

The initial eval is a small directory of seven files:

haiku/<br>├── eval.yaml<br>├── tasks/<br>│ ├── pelican.yaml<br>│ └── otters-in-love.yaml<br>├── configs/<br>│ └── default.yaml<br>├── graders/<br>│ └── default.yaml<br>├── checkers/<br>│ └── three-lines<br>└── run-llm

Improving...