The Boring Frontier of Robotics

The Boring Frontier of Robotics

Why backflips are easier than folding laundry.

I recently had the pleasure of spending a week at the Royal Society’s Summer Science Exhibition, helping out at an exhibit hosted by my PhD lab (the A2I lab at the Oxford Robotics Institute). Over the course of the week I delivered a spiel about our work countless times to members of the public. I thought I’d note it down for posterity.

Yours truly at the exhibition, doing a wide thumbs up in honour of Frank, the lab's wide-armed robot.

The Spiel

Have you seen those internet videos of robots backflipping, or running marathons? What’s surprising to a lot of people is that those are easier for robots than basic things we do daily without thinking about it — things like making a cup of tea. Think about it; how come AI beat Garry Kasparov at chess 40 years ago, yet no one sells a laundry-folding robot? It’s because many things we find hard (like chess) are easy for machines, and things we find easy (interacting with the real world) are unintuitively hard for them.

This backflip is attention-grabbing, but in many ways it's simpler than things we do without a second thought.

The reason we find it easy to do stuff in the real world is that our brains are incredibly well-evolved for the job. Consider what happens when you go to pick something up. Before you move a muscle, your brain has figured out:

Where the object is in 3D space, relative to you

What size it is

What shape it is (including the bits you can’t see — we regularly place our fingers on the backs of things we pick up)

How heavy it’s likely to be (and hence how hard we should grip)

How grippy the surface is

How we should hold it for the next thing we want to do (am I picking up scissors to cut paper or pass them to someone?)

How our muscles should move together to get the hand to its destination

How our stance should change to enable us to pick the object up (you might widen your stance or lean when grasping something heavy)

How the object might move once we pick it up (e.g. is it solid, or does it contain a liquid which will slosh around?)

And that’s just picking something up. We do all of this effortlessly, day in and day out, but robots get none of it for free. That’s why LaundryBot isn’t on supermarket shelves yet.

Our exhibit focussed mostly on manual assembly — a task called RAMP which would take a 6 year old a minute or two. Our robot, Frank, managed it in maybe 5 minutes, so not too bad, but far from superhuman. Part of the reason for the non-superhumanness is that we were shooting for reliability. We were at the Royal Society for a week straight, running the demo on repeat. Robotics and AI researchers will hit a 90% success rate and call it job done, but a one-in-ten failure rate is catastrophic when you’re demonstrating to sceptical teenagers. Modern robotics (and LLMs, for that matter) falls short of the reliability we’d want for useful autonomy in the messy real world — another reason LaundryBot isn’t here yet.

In short — robots probably aren’t coming for your job, or coming to kill you or anything that dramatic. They might be able to backflip on TikTok, but they still can’t make you a cup of tea. That’s what we’d like to solve.

FAQ

A fun aspect of chatting to so many people over the course of the exhibition was being asked questions. Many questions. Here are some that came up repeatedly, and the responses I settled on.

Haven’t robots been doing assembly in factories for decades?

They have! The reason is that you can control everything in a factory. If I’m a hole-punching robot in a factory, I don’t need to know anything about the world. I get given a chunk of steel, and I move from position a to position b every time. Forever. If my chunk comes in the wrong orientation, I can’t really adapt on the fly. So in factories, we get the benefit of robot control without the need for intelligent decision making. Outside of factories, we’re not so lucky — our metaphorical chunks of steel rarely come the same way twice.

What about surgical robots?

Surgical robots appear different to factory robots, but they actually reinforce the general point. Surgical robots provide fine motor control, but they still delegate real-world understanding and decision-making to a human operator. The robot doesn’t need to know where to make a cut, or which organ it’s dealing with: it relies on the surgeon to fill in for its lack of understanding.

What about ChatGPT? This must nearly be solved?

ChatGPT is really impressive. What’s exciting for AI researchers is that it wasn’t trained for specific jobs, like writing poetry or knowing the capitals of countries. Instead, it learns all that stuff from a simple task — next-token prediction of internet data. ChatGPT suggests a path to training robots that can generalise to new situations.

So why don’t we just train ChatGPT for robots? Data . Us humans have spent decades writing things on...