Through the Agentic Looking Glass — Alex Self<br>Through the looking glass I went and what awaited on the other side revealed wild possibilities. Running a Heraclitean lens that performs a unity-of-opposites analysis on a growing middleware stack’s tensions, shifted a bit of my understanding of how LLMs can work. Cognitive operations, intellectual traditions, uniquely strange behaviors and the fine line of the absurd have been explored in building agent definitions.

What has been discovered through practical assembly is a multi-layered structure in how LLMs interpret information, with each layer becoming a bit less deterministic the further you go. The more I’ve worked with these systems, the more the little quirks of language have unveiled a new frontier.

One of the driving motivations of this deep dive has been figuring out how to improve the definitions that drive the agents; the other, is that each layer found ended up revealing the next and I wanted to continue the pursuit until the layers bottomed out. The sum of which shows how LLMs can unlock cognitive functions at various levels of thought.

In order to build an understanding of what is possible with the various agent definitions, I attempted to classify what they are and at what level they operate, thought wise. A six agent type taxonomy was created that can be applied at each layer: Analyst, Executor, Explorer, Forecaster, Generator and Validator. Two writers: Executor and Generator; four readers: Analyst, Explorer, Forecaster and Validator. One axis to describe what they are and one that describes the layer they act on.

This post takes a walk on the read-side. Through we go.

Table 01Agent Types and LayersReadersWritersLayerAnalystExplorerForecasterValidatorExecutorGenerator1Structuralbuiltbuiltbuiltbuiltbuiltbuilt2Behavioralbuiltbuiltbuiltbuiltopen territoryopen territory3Reasoningbuiltbuiltbuiltbuiltopen territorybuilt4Epistemicbuiltbuiltbuiltbuiltopen territorybuilt5Meta-analysisbuiltopen territoryopen territoryopen territoryopen territoryopen territorybuiltopen territory

Layer 1 - the artifact’s structure

The first agent definitions I created consisted of the Code Validator and Test Architect. The builds that followed began to assume a similar classification: bounded by somewhat of a mechanical role or profession type. Software Architect, Developer Experience, Privacy Officer. These definitions contain the patterns that focus on the structure of artifacts. They operate on the artifact directly and are grounded by questions with fairly deterministic answers: is this SQL safe, does this code validate, does the functionality described in the README actually work? These agents can be effective in their roles for what they do and they also miss angles other layers would cover during an audit.

This idea calls back to a concept I’ve been working on: the cognitive parallax. Where the difference between multiple view points helps provide a clearer understanding of the observed artifact. The divergence between the views is the measurement.

Agent definitions within Layer 1 discover structural, semantic and pragmatic aspects of an artifact. Mechanical bug fixes, refactors, documentation gaps and occasional design decisions on architecture or component functionality. The findings these agents produce are oftentimes fairly straightforward to address, as the depth they typically operate on is more or less on the surface area. Sometimes the agents will surprise with a deeper dive, but generally, these definitions are focused on a matter-of-fact type of analysis and validation. Moving further, we get to a different layer and one where dynamics are examined.

Layer 2 - the artifact’s behavior under time and stress

The next set of definitions I developed focus on the artifact in motion. These agents ask different types of questions: What does this artifact do under stress, over time and across state transitions? Every definition in Layer 1 that I’ve built almost entirely analyzes a static state. The Layer 2 behavioral agents are actively seeking to understand how state mutates shape or transitions across services, how resilient a database or API is to throttling, how a system would handle malformed shapes, fault injections and all the other ways that can break a system.

A couple of the agent definitions I’ve built to test out this behavioral layer: Runtime validator, State validator, Performance validator, Chaos validator. The findings are often sharp and reveal interesting aspects on the behavioral quirks that typically manifest during system stress. In Layer 2, you could have a Chaos Validator, Explorer, Analyst and Forecaster, all of which audit the artifact in motion, but from slightly different angles and responsibilities. The layer does not change, but the core operation and axis the agent works within does.

An Explorer operating within Layer 2 maps where states transition, areas of resilience (or questionable ones); a Forecaster can take an...