arda tasci

arda tasci

content

about

talk to me

Towards a Harness That Can Do Anything

I've been thinking about how to free LLMs from the chat pane for a few years now. Watching what people have tried and seeing what worked (and didn't), I've formed my own opinions on the right way to approach it. Here are my thoughts, and what I've been working on recently.

What Makes a Good Harness

I. Should be naturally intuitive to the Agent.

II. Everything should be transparent so that the Agent can self-develop or heal (or audited postmortem).

III. Must be as lean as possible and flexible.

IV. Error survival, update survival, no memory corruption or degradation over time.

With the advancing intelligence of LLMs, harnesses will eventually be reliable.

The real matter at hand is reducing how much cognitive load (measured in tokens) you are putting on your bot.

Preliminary Truths

The most important things we've learned over the years.

Determinism as much as possible. The LLM should choose what goal to pursue, but the deliberation towards that goal should be well-defined or at least a collection of well-defined steps.

The core prompt should be as small as possible, and the LLM should then choose what skills to load into context at runtime.

LLMs start going crazy as you approach context limits.

Don't Play the Odds, Play the Bot

A good harness MUST make use of the a priori coding knowledge of the LLM. Coding and systems administration are heavily overrepresented in the LLM's training data, so give it an environment it is already comfortable in; wrangling it through a novel one ultimately wastes tokens. Similarly, precious context should not be wasted on things like file discovery, traversal, etc. -- good harnesses make delegation easy and efficient. At the same time, a harness should feel light to the LLM, but actually do a lot of things in the background. These include logging, sanity checks, failsafes, sanitizations, etc.

Auditability, Logging, and Self-Healing

Everything is vulnerable, and all agents eventually fail. There are two types of Agent failures:

LLM Level

Harness Level

Failures at the LLM level cannot be directly patched, but the risk of such a failure can be mitigated by the harness.

Harness-level failures can be recovered from, and should be fixable at runtime due to the turn-based nature of the LLM.

In order to fix a defect, the Agent needs two things: great logging, and a clear error message.

A Unified Data Layer That Can Do It All

Most of these requirements are age-old questions; only the verbiage has shifted from Users to Agents. So it's worth asking: what can we learn from the before-fore times, when people used to actually write code?

My Hypothesis: the Unix / Linux Environment is a natural candidate, and could be turned into an Agentic harness with a few modifications.

I am, by no means, an expert on Unix or any of its descendants, but I have spent the last decade learning about its history, design choices, and usage. A lot of it maps beautifully onto our predicament; plenty of it doesn't. Think of U/L as a motivating analogy rather than a direct comparison.

Mapping Concepts of Old to New, and the Unix Philosophy

In case you might have forgotten, or not seen it before, here is the creed of our forefathers Ritchie and Thompson:

Write programs that do one thing and do it well. To do a new job, build afresh rather than complicate old programs by adding new "features".

Write programs to work together. Expect the output of every program to be the input for another program.

Write programs to handle text streams, because that is a universal interface.

These perfectly encapsulate the issues with harnesses today: they're overly complicated, attempting to do a lot of things, and the trajectories our Agents are expected to take are often ill-defined. These are symptoms of the same issue: there is no way for the Agent to hold agency over itself. In many cases today, tools are directly loaded into context, and system prompts are preemptively filled with warnings and specific rules and guidelines that developers have (which eventually fade turn-over-turn).

We can then derive our own set of principles for designing our harness:

Write modular, transparent tools that do one thing and do it well; ensure they fail loudly.

Write tools, skills, and connectors that work together. Skills dictate workflows, tools are the means to execute them, and connectors are the data which the Agent manipulates.

Text streams are a universal interface, and a Language Model has home-court advantage. Everything should be a flat text file.

Ambiance: My Take on Harnesses

Everything is a File

Have you ever manually done JSON wrangling? What about constructing heavy curl commands to query an endpoint? Complicated regex? I haven't, because they are a pain in the ass. An LLM might have an easier time with them than you, but rest assured that they prefer plaintext too. Thus, whenever dealing with...