Why Claude Uses the Browser Like a Drunk Intern<br>Skip to content

Back to BlogAI AgentsBrowser AgentsAutomation<br>Why Claude uses the browser like a drunk intern, and how to fix it<br>Korbinian Abstreiter, Co-Founder of pluno.ai<br>July 14, 20268 min read

How we accidentally built a browser agent

We spent the last few years building AI agents for customer support.

The first version was basically agentic search over support context: docs, tickets, prior resolutions, internal notes.

That gets you surprisingly far, but support work quickly runs into product state. The answer often depends on the customer's account: configuration, permissions, billing state, or some admin setting buried three screens deep.

So the agent has to inspect the product, and sometimes change it.

We didn't start with "let's build a browser agent."

We started with "the agent needs to be connected to the API."

Why Claude uses the browser like a drunk intern

You've probably used at least one browser agent by now, whether it's Comet, Claude, ChatGPT Atlas, Browser Use, or anything else.

The loop is always the same:

Explore the page. Maybe even take a screenshot. Find the likely target. Move the mouse. Click. Wait. Screenshot again. Scroll. Try a dropdown. Recover from a modal. Guess whether the state changed.

I get it - this is the information we have, and the functionality that's exposed. But it's horribly slow, and highly inefficient.

It's like asking a voice agent to book a barber appointment by calling another voice agent. Technically possible. Maybe even magical the first time. But absurd once you realize both systems could have exchanged structured data directly.

The UI is optimized for humans: layout, labels, progressive disclosure, hover states, responsive breakpoints, visual hierarchy. None of that is the product's actual execution model.

Underneath, the frontend is already speaking to a structured system. A click becomes a request, mutation, RPC call, websocket message, server action, signed request, persisted GraphQL operation, or some product-specific command. There are inputs, outputs, errors, permissions, side effects, and state transitions.

That's the interface an agent should want, and it's exactly why we have MCP servers and CLIs.

But since many products still don't have an MCP server, we're stuck with agents looking at a screen to figure out which button to click.

The obvious solution

Agents should use the structured interface underneath the UI, not pixels.

One operation can replace a dozen UI steps. A small script can replace a fragile click path.

Instead of asking a model to infer intent from screenshots and layout, give it objects, fields, methods, responses, errors, and invariants.

That's a much better fit for current models. They're already extremely good at code, structured data, and tool calls. They're much less reliable when the task is "interpret this screenshot, click the right thing, and hope the UI didn't shift."

Pixels are a lossy transport for behavior that already exists in code.

So the question seems easy: why not use the underlying API directly?

The not-so-obvious solution

The hard part isn't that the interface doesn't exist.

The UI uses it constantly.

The hard part is that it is not structurally exposed or documented.

It's usually not public. It might not even be stable. It's often split across REST endpoints, GraphQL operations, websocket messages, signed requests, WASM modules, web workers, client caches, server actions, and frontend-only conventions. Some of the important behavior may live in request sequencing, id resolution, validation responses, cache invalidation, or permission-dependent branches.

"Just use the API" skips the part where the agent has to learn the actual contract the UI is relying on.

The agent needs to know:

what objects exist

which operations create or mutate them

how authentication works

how server-side language maps to the user-facing UI language

heck, sometimes even that status ids depend on the user's language (*cough* Zendesk)

That information is often present only implicitly: in frontend code, network traffic, runtime state, cache updates, and error handling.

So the browser still matters.

Not as the runtime interface.

As the discovery interface.

When the frontend performs an action, it reveals the product contract. It fetches objects. It sends operations. It validates inputs. It receives errors. It updates state.

An agent can check the source code, observe what happens, and extract the shape of the workflow:

routes, operations, or message patterns

request and response shapes

required fields and object relationships

validation, authorization, and success states

So how can we make an agent learn from that?

By building skills.

A simple loop already works quite well: give an agent its own sessions, then tell it to update its skills based on what worked, where it struggled, and what it newly discovered.

Since this learning happens across...