The Dark Arts of Web Automation

Ominous, right?

Sounds like I'm about to teach you something you'll need a lawyer for.

…We'll come back to the lawyer in a bit.

Here's a little bit of background…

So…

I was getting ready for this talk -- and OpenAI threatened to ban my account ...

Just for the work that I was doing in preparing it…

I checked my email the other day, and I found this.

What a shocker this was!

So -- what does one have to do to earn the banhammer?

For cyber abuse?

With a web browser?

This. This is what I was doing.

Every one of these is being solved by an AI agent using a web browser

... and in 15 minutes or so, you'll understand exactly how.

I want my agents to be able to use the web the way that I do -- book the things, send the emails, fill in the forms -- so I don't have to.

But, the moment something that isn't a person starts clicking, many webpages fight back.

So -- this is a talk about winning that fight. And it starts with one slightly unconventional idea.

Here it is. The premise of the whole talk.

'A CDP browser is just like a meatbag with a mouse.'

...

At least as far as Google, Cloudflare, and the rest can tell

No joke -- if you have your agent drive a browser through the Chrome DevTools Protocol, your agent's clicks and keystrokes travel the exact same path inside Chrome that yours do.

That's the big idea. The rest of the talk is how you pull it off. And that comes down to three things.

A CLI -- not an MCP.

The Chrome DevTools Protocol -- which is where a tool I built, called chrome-agent comes in.

And a loop, run on a ladder.

So, let's start with the one that picks a fight.

First thing: give your agent a command line interface (shell-based tools), not an MCP server. And before some of you knuckle-up on this debate, there are specific reasons for this.

It's worth noting that capability is a wash -- CLI and MCP both got to the right answer about 83% of the time in a recent study by Arize AI.

HOWEVER, a CLI beats an MCP in reuse, speed and cost.

Reuse first. A CLI sequence can be programmed: write it once, run it a thousand times, without a model in the loop. Whereas MCP hits the model on every turn.

The CLI is FASTER for a similar reason -- because there's no model in the middle of every step. In that same study -- MCP took 71 round-trips and 8 minutes for the same task that was completed with just 7 calls and under a minute using the CLI. Hold onto this one -- speed comes back at the very end.

And token cost -- Anthropic reported that executing code instead of running MCP can be 75x cheaper in terms of token usage.

So -- what are we actually running on the command line?

We drive the browser with the Chrome DevTools Protocol -- the second thing your agent needs to appear human.

You already know of this protocol, even if you've never heard its name. That developer panel that opens in Chrome when you hit F12 drives the browser using it. Your agents can speak it, too, using the chrome-agent tool. Chrome Agent also makes it easy for your agent to write code to replay CDP interactions.

The surface area of CDP is enormous. 57 domains, hundreds of methods and events, covering everything the browser can do. I've grouped these domains into 8 buckets to make it easier to hold in your head.

But the good news is that you don't need all 57 domains. To interact with a page like a human, you usually need just a small subset of these.

It's easiest to think about the subset of CDP domains you'll frequently use in terms of the 'digital senses' they provide your agent.

You SEE the page -- read its structure from the DOM, its semantics from the accessibility tree, or just take a screenshot when you want the pixels.

You HEAR what the page reports about itself -- the network data, its console, and its logs.

And you OPERATE the page -- with clicks, keystrokes, and navigation.

The third and final thing you need for your agent to appear human when using the browser is what I call "a loop on a ladder."

So here's the loop:

Sense, act, verify -- and you repeat it until the page gives in.

Sense -- perceive the page through one or more channels -- the DOM, the accessibility tree, a screenshot.

Then Act -- do ONE thing -- click something, type something, select something.

And then Verify -- sense again, but through a different channel than the action. e.g. If you clicked something, don't ask the click whether it worked -- look at the screen, or the network instead.

Sense where you are. Make one move. Confirm it landed. Then iterate again.

And when the loop won't close -- so when you sense, act, verify, and the page still won't do what you want -- that's the page fighting back and telling you to climb the ladder.

The Meatbag Ladder, that is! This is a ladder of techniques that are increasingly more human as you climb it.

It has 3 rungs and you climb only as high as the page forces you. Said differently, you climb to the lowest rung that works.

On Rung 1 you...