Spec-driven skill generation←Back to blog<br>I've been synthesizing the agent skills using skill-creator for a while and I was usually happy with the results. The models have gotten smart enough where they can capture the intent pretty well and agents can usually follow the instructions well.

The problem for more complex skills is that I'd like to know for sure that they still work over time, or that I can change them and still trust the skill is good. I want them to be a testable unit , derived from actual intent, or set of rules, and behaviors I mean for skill to follow.

A good example of that is dotagents, Sentry's library for managing agent tools locally. It comes with a skill pre-bundled. Over time that skill has drifted from the implementation . The CLI commands got expanded, the set of features it supported changed, the skill didn't.

I've been experimenting with various approaches on this and landed on an interesting version of that. Inspired by OpenSpec I've implemented a tool, called Skillet that focuses on generating skills from a markdown spec file. The spec file is responsible for capturing following signals:

the intent (why does it exist? what should it be used for?)

the behaviors (agent should do X when Y happens)

the constraints (agent must/must not do X)

the triggers (when should the skill be used)

That spec file is a source of truth. The prose of SKILL.md is generated from it. The list of behaviors/triggers the spec describes translates to eval cases that skillet can then run for in your harness of choice.

That's it. It's a simple format, nothing fancy, kept in a readable markdown file. But it gives you just enough structure to make your skills testable, and more maintainable.

How does it work?

skillet spec management is inspired by OpenSpec.

The spec.md file that is generated from skillet is a readable artifact, structure of which is opinionated and strict. skillet CLI is an agent-first tool that manages that spec file. The agent interacts with skillet to modify it, verify it, and run evals. The CLI itself doesn't use any LLMs directly so there's no setting up of any API keys, it just leverages your agent.

Because it's meant for agents first, it's also extremely easy to try if you'd like to give it a spin. All you have to do is prompt your agent:

Install the skillet-authoring skill globally from https://github.com/getsentry/skillet.

and it will add a skill-creation skill that will trigger when you need a new skill. That's it - very opaque for the developer.

Here are some examples of skills created with skillet:

Dotagents skill

An experimental rewrite of Effect skill by @kitlangton

Garfield, the local review skill by @zeeg

But does it work?

Skillet is experimental and I mean it. I think it's valuable and I've enjoyed iterating on skills using it myself, it's a simple tool and it does what it promises.

The main value proposition for skillet is to test and maintain the skills. Make sure that the behaviors you want and expect from your skills are captured, and can be changed in a slightly more predictable manner. I'd say for that - it works great.

Skillet is not a tool to raise the accuracy/fidelity of the skill! I've run a few benchmarks comparing skillet generated skills to one shot skills generated by models through a single prompt, and I don't see the accuracy to be improved. Models have gotten smart enough to close that gap very well.

There's also a variance argument at play. Technically it is your agent that's generating the prose based on the spec. This is a non-deterministic process. If you run skillet generation 10 times, you'll get 10 different skills - some will be slightly better than the others, and skillet isn't worrying about it. If it passes evals, it's good enough.

Summary

If you're interested in trying skillet, just ask your agent to install it

Install the skillet-authoring skill globally from https://github.com/getsentry/skillet.

It's a very hands off tool. If you have any feedback, let us know!

githublinkedintwitterrss