Add telemetry and observability to your AI agents

The Davenporter

SubscribeSign in

WIBTM: Add telemetry and observability to your AI agents<br>Telemetry and observability give you the means to start capturing data that make your agent systems long running, and improve their capabilities over time.

Jason Davenport<br>Aug 18, 2026

Share

Most agent examples start with agent instructions, a model, and a few tools, and end there. As a part of a talk I gave at AI4, I tried to create a few agent examples where you can start playing with ‘state’ for an agent in different ways.

This blog is essentially a recap of my talk and interesting things that I learned.

Why do agents need telemetry, and specifically traces?

In application development, traces that I created in the past were typically viewed as relatively expensive telemetry operations to be used in the event of failure. If you had actually implemented tracing, then you could find which part of a multi-layer system broke.<br>In agent development, deployment, and monitoring, traces are important to measuring the system performance and keeping the system well tuned. They are not just for finding failure (although they’re good for that too).<br>In agent telemetry, a trace is commonly referred to as a trajectory. A trajectory represents the steps an agent took based on an input to meet an output. A single trajectory may represent a few traces or spans, but it encapsulates all of the agent logic and decision making. This ultimately means that a trace should also capture all of the prompts and responses within the agent’s reasoning loops.<br>Telemetry is a powerful tool for us. We can use it for offline analysis to improve our agents, create memories using dreaming, and perform real time analysis which may include shutting down a discussion if safety parameters are violated over the course of a current trajectory. However, telemetry isn’t necessarily a real time input. There are other tools we can use for this.<br>Other methods and tools we need to capture and use agent state

Most agent examples rely on a simple, in-memory session service (or no session service by simply appending prompts and responses). These are fine for examples, but they lack durability when you start to run agents over time. For example, how would you manage:<br>Connection failures between the agent and the user

When a user leaves a session but wants to pick up a session later

Storing information securely for an agent over a session that the agent may not necessarily have access to

State is incredibly important in agent design. For those working in data, it’s a tough problem. Managing state over time presents a number of questions like ‘when should the state be reset’ or ‘what state should be saved for even longer’?<br>For session state, there’s typically a database involved. Databases like Redis / Valkey are great for fast, in memory session management. The agent can write to these whenever it needs to, and you can set a Time To Live to automatically expire information that’s no longer relevant.<br>You can also use a database like Postgres or Google Cloud Spanner for session management. While it’s not quite as performant, you gain other features like being able to store short and long term memory together, and even tools like graph commands to give your agents more access to information.<br>Either way, you want to add one of these methods early to an agent so you can understand how it impacts the agent’s behavior as you start to launch.<br>An example using an agent that is writing SQL over a long running session

I built a simple agent design that demonstrates the impact of state on agents. It’s written using Google Agent Development Kit (ADK), Cloud Run, and Cloud Spanner as the database.<br>This playground gives you a way to compare and contrast agent behavior. You can use two agents together to see what happens when one agent has access to a database to store sessions over time.

Here’s the code, and here’s how the big pieces work.<br>Session management

The session service is attached to Google Agent Platform Session service, or Cloud Spanner. You can use any database here though. The point is that the framework, in this case Google ADK, uses the database to log session information so it can be retrieved later. It’s pretty straightforward. Most of your app logic here is focused on when a session should be resumed (if automated), and then how to manage deletion. Here’s the block that creates the Agent Platform Session service. You can use this managed connection or build your own.<br>class VertexAISessionService(VertexAiSessionService):<br>"""ADK SessionService backed by Google Cloud Vertex AI Agent Engine Session Service.

Provides managed multi-turn session persistence using Vertex AI Agent Engine.<br>"""

def __init__(<br>self,<br>project_id: str = DEFAULT_PROJECT,<br>location: str = DEFAULT_LOCATION,<br>agent_engine_id: Optional[str] = None,<br>express_mode_api_key: Optional[str] = None,<br>):<br>proj = project_id or DEFAULT_PROJECT<br>loc =...