The First Robot Trained Entirely on Dog's Video

Postcapitalist Robots

SubscribeSign in

The First Robot Trained Entirely on Dog's Video<br>I taught a robot to behave like a dog, using a phone video (dog2go2 I)

Postcapitalist Robots<br>Aug 17, 2026

Share

A quadruped robot fully focused on the TV, learning from a dog. Image generated with Ideogram 4.0, locally in my RTX 3090. Design by @gazebyputu.<br>In the era of robotics, where robots are starting to dance like humans or run in the middle of the forest like pumas, we shouldn’t be mere spectators, we have to own our robots.<br>Owning our robots? What do you mean?

Yes. Common ownership.<br>There are many ways of common ownership, and that’s a big topic —a central topic in Postcapitalist Robots.<br>The idea is shifting this late-stage-capitalism paradigm, where we are simple consumers, and start to be co-owners and co-producers of the robots.<br>Broad robot ownership can be achieved through public property of the means of production —of course, with a government aligned to the public interest, and not just the interest of a few—, through cooperativism, and through many other ways yet to be conceived.<br>Ownership means you have control over the outcome.<br>You feel sovereign. You feel that if something is not working, you can change it.<br>Please, note that I’m talking about broad ownership. If we talk about just a few people’s ownership —i.e. stakeholders—, of course, a simple private company does the trick.<br>Another way of gaining ownership is through education and accessibility to resources. If we know how our robots work, and if the knowledge, tools and infrastructure are publicly accessible and affordable, then we are closer to owning our robots.<br>And that’s the power of open source, open weights in AI, and open-source robotics. We know how things are built, we can built on top of them or play with them. We can interact with them, improve them, fix them, repurpose them.<br>We don’t wait for a company to create a subscription that we can buy —in case we can afford it. We just build it.<br>Today, I answer the question:<br>What if we could teach a “robotic dog” —quadruped robot— to imitate what a real dog is doing in a video?

What if everyone could do it because it’s open source?<br>Let’s do it —below, in the video, you can see what I mean.

If you really want to learn more about robotics and how to own them, consider subscribing!

Subscribe

If you are here just to explore the code, and to make it run. Please, jump ahead, it’s completely open source:

Code here<br>One-Shot Learning.<br>The Holy Grail of Robotics.

Humans —and me being in the top-1000— can be very clumsy.<br>But there’s one thing humans are unbeatable at: humans can learn many complex tasks, by just seeing one example.<br>The current state of the art in robotics tells a different story. Current robots need millions of trial-and-error examples to learn a simple task, and they perform it veeeery slowly.<br>Normally, we talk about “years of experience” —the amount of time the robot needs in order to learn the downstream task. For instance, some examples required 100 years of experience to learn hand dexterity —of course those 100 years are compressed into tens of hours through the speedup of simulation time in computers and parallelization of many computers with many different scenarios.

Humans just need one visual example, and many tasks can be easily mimicked. Of course, one might argue that we had millions of years through evolution, as well as the years we had when we were babies, to be able to do that. That’s also true, and it is still an open question about intelligence.<br>Others, like Yann LeCun, argue that babies spend years just watching the environment in order to develop a world model internally. Afterwards, they just use the world model to interact with the world, but in a predictive way. According to him, robots are just lacking this world model, and the million years of evolution aren’t necessary.<br>For the sake of this article, we’ll use a video as an anchor, and we’ll train a reinforcement learning policy to follow dog’s behaviours. Even if we just use one video example, the policy has to see the behaviour thousands of times before being able to mimic it adequately.<br>The robotic dog learns from the real dog.<br>Just by watching.

I’ve spent the last month working on this project. It’s very exciting to see how a robotic dog can learn a behaviour from a real dog in simulation (with full physics), using just a video as input.<br>Be aware that there are many ways of achieving this. This is just one proposal, based on my experience in the field —my PhD was about sim-to-real for quadrotors—.<br>The idea is the following:<br>Using just a monocular video (a video taken from a single camera).

Extracting the 3D pose of the dog skeleton in every frame.

Retargeting the trajectory of the real dog joints to the robotic joints.

Performing Inverse Kinematics (IK) —to know which joint states would lead to a specific trajectory.

Training reinforcement...