I gave Claude a robot body - by Tal Raviv

SubscribeSign in

Playback speed

Share post

Share post at current time

Share from 0:00

0:00

18

I gave Claude a robot body<br>If an LLM can click around my computer, it can drive around my house.

Tal Raviv<br>Jul 28, 2026

18

Share

It started with Claude’s computer use feature.<br>I was using Claude Code to improve Familiar when it took over my screen and started clicking around. I got curious and dug into the tool calls, only to discover that it’s just “screenshots in, mouse coordinates out”:1

The chat thread went:<br>Claude took a screenshot

It called a tool to left click (or right click) “the pixel at x, y”

It took another screenshot

etc.

It threw me back to programming in Logo as a kid. Logo let me move around a turtle (either virtual or a two-wheeled robot named “turtle”) with text commands:2

So LLMs can read images and print out text commands…<br>And I can control a robot with text commands. And that robot has a camera…<br>…so an LLM should definitely control a robot!<br>I promptly ordered a robot kit with 1) a camera that 2) I can control from my computer terminal. Once everything was hooked up, my Claude thread looked a lot like the computer use screenshot above:

The result:

Assembling the robot

I put all my lessons learned for hardware, software, and skill, etc. up on Github.

I named it Bubbles, because “there’s no angry way to say bubbles.”<br>The hardest part was the stupid logistics: sit down, make coffee, realize I'm missing a random part, order it, wait, repeat. Once I got through the docs and had terminal access to my robot, I handed the reins to Claude Code and did everything through chat.<br>It was super clumsy, and impressive that it worked at all

Out of the box, I found myself with a snarky, token-inefficient Roomba that doesn’t clean.<br>I severely edited the video above for internet attention span. In reality, Bubbles paused a long time between talking, listening, moving, snapping photos, etc. (even on smaller, faster models).<br>I asked it to locate the door to the room and park exactly 20 centimeters from it (it has an ultrasonic distance sensor). What followed was fifteen minutes of it confidently parking in front of a filing cabinet.<br>My voice transcripts:<br>Me: “Almost right, you’re not facing the door, you’re facing the contents of my closet.”<br>Me: “I didn’t mean closet door. I meant the actual door to the room. Please zoom out. You’re kind of stuck in this corner and you’re nowhere near the objective.”<br>Me: “No, this is absolutely not the correct position. This is the closet. The door to the room is not the closet.”<br>[Bubbles did a 360 degree scan of the room, but swung too fast and didn’t snap a photo of the door, and returned to face the closet]<br>Me: “Honestly, it’s like 45 degrees to your left.”

Spatial reasoning is hard. AND I’m impressed that any of this worked out of the box. There are entire companies built around AI for the physical world: Physical Intelligence, World Labs, Field AI - and strong opinions about whether LLMs can reason about the physical world at all. After one day with this robot, I get why.<br>That said, there’s also companies taking open-weights language models and post-training them to be good at lab science (highly recommend this episode). Just because it’s language in/out doesn’t mean an AI system is limited to thinking in language.<br>So yes, language is clunky, but what if an LLM could make this work? What if an LLM could map out a house in language, CSV, excalidraw format, XML, who knows? What if we could all have robot butlers powered by cheap LLMs?<br>Then it started learning

Since Bubbles had a file system, it could record memories as text files and saved photos, too:

Here’s the notes it made about me:

I’m glad I made a good first impression.<br>It started to learn its own body, for example not to fully trust the ultrasonic sensor when approaching human feet:

When something worked, Bubbles documented it to save time in the future. It started off over-shooting its turns, and by the end of the day taught itself to be more gentle. It also documented how to do a three-point turn:

Calendar, email, skills, scheduled tasks, and memories

Since I was running the robot using Claude Desktop, it had all the tools and skills I use daily. I could ask what time my flight was tomorrow, or have it summarize my email.<br>Then Bubbles suggested it could patrol the house and publish a field journal of its observations, and did a test run: https://bubbles-patrol-test.surge.sh/

Laurel was out the whole day, so Bubbles had never seen her, and I never mentioned her. When it saw her, it immediately deduced who she was: “Well well well. A woman on a stool, filming me…. Am I speaking to Laurel?”<br>I assume this was either thanks to Claude memory, or because it just looked up my calendar. It then made a note:

Other memories included: “Owner’s flip-flops (teal straps) are a reliable owner-marker at my eye level.”<br>I have a feeling that if I just keep using...