My AI Agents Were Working, but I Had No Visibility

SubscribeSign in

My AI Agents Were Working, but I Had No Visibility<br>This report shows how I integrated the agents behind Applied into LangSmith, an open-source observability framework, to track traces, token spend, and performance

Hugo Santana<br>Jul 07, 2026

Share

Back in May, I wrote about the 6-agent system behind Applied .<br>The setup was simple but useful. The agents researched AI adoption use cases, tools, and vendors, then categorized everything into the Applied living map. There was also a translator agent, which made the information available in Spanish, and a match maker agent, which connected users with relevant content based on their preferences.

Applied’s Agentic Architecture v1.

How I Built an Agentic Research System<br>Hugo Santana<br>May 6

Read full story

Since then, the system has grown. I added agents for open and closed model research, adoption signals from Hugging Face, OpenRouter, and job boards, plus a weekly digest agent that uses the match maker to send more personalized reports.<br>The agents were working. They were researching, extracting, classifying, translating, and turning messy AI adoption signals into structured information. I could see the final outputs and read the logs, but I could not clearly understand how the work was happening below the surface.<br>I wanted to answer simple but important questions.<br>Which agents were spending the most tokens?

Which tasks were failing?

How long did each step take?

That is what led me to LangSmith.<br>In this report, I’ll walk through how I integrated the agents behind Applied into LangSmith, what I can now see across the system, and why I think observability is only the first step toward improving multi-agent products.<br>Subscribe to The Applied Report to receive insights on AI Adoption

Subscribe

What’s LangSmith?

LangSmith is an observability and evaluation platform for LLM and agent applications, built by the LangChain team. Despite the name, you don’t need LangChain to use it. LangSmith is framework-agnostic, which made it useful for Applied, where my agents were already running through a custom setup.<br>In short, LangSmith turns agent activity into traces. Each execution becomes a run with inputs, outputs, duration, status, token usage, cost, errors, and nested child steps. Instead of digging through scattered logs, you get a searchable history of what happened, how long it took, how much it cost, and where it failed.

The broader LangChain ecosystem has three main pieces: LangChain, LangGraph, and LangSmith.<br>That matters because once agents start doing real work, the question is no longer only whether they completed a task. The question becomes whether they completed it efficiently, reliably, and with the right level of quality.<br>In practice, LangSmith makes it simpler to trace and monitor runs, evaluate outputs, use annotation queues for human review, manage prompts outside the main codebase, and set alerts for cost spikes, latency issues, or rising error rates.<br>Why the Integration was Easier than Expected

LangChain, LangSmith, and LangGraph are open source and API-first, which makes them easier to use with AI coding tools like Claude Code. Their public documentation, accessible SDKs, and clear concepts give coding agents enough context to inspect, explain, and implement them.<br>Claude Code was able to understand LangSmith, help set up the integration, and guide me through the parts of the product I needed, including where to configure API keys in the UI.<br>As I wrote in my previous post , this is one of the advantages of open-source and API-first products in the agent era. They lower the barrier for developers, while giving agents enough context to reason about, implement, and extend them.

Open Source, APIs, and the Rise of Agent-Led Growth<br>Hugo Santana<br>Jun 26

Read full story

How Data Flows into LangSmith

There are a few ways to send traces into LangSmith. The Python or JavaScript SDK can wrap LLM calls, run records can be sent manually through the REST API, and OpenTelemetry can be used when an application is already instrumented that way.<br>For Applied, the important part was that LangSmith did not require rebuilding the agent system around LangChain. The agents keep running exactly as before. A lightweight step was added so that, once a routine finishes, a summary of what happened is sent into LangSmith as a trace. The trace measures the run without changing how the agents do the work.<br>Two things make that summary useful. First, each agent marks its own stages as it goes, such as “starting Phase 1” or “finished Phase 1,” so LangSmith can show the run step by step instead of as one opaque block. Because these markers live inside the agent’s instructions, changing or renaming a stage is a small edit in one place.<br>Second, the summary includes the numbers that matter for observability: how much the run cost, how many tokens it used, how long each stage took, and which model ran it.<br>Open...