A Robot Hand That Taught Itself to Play Piano Could Change the Future of Machines - USC Viterbi | School of Engineering

(Image/Midjourney)

In a neuro-robotics lab at the USC Viterbi School of Engineering, a small mechanical hand heard a melody and played it back.

No weeks of training. No massive datasets. Just two minutes of random doodling on the keys—like any child would.

The hand got so good at playing that it “auditioned” before two musical judges who listened to its performance, blindly, alongside those of four human pianists.   The judges sometimes couldn’t distinguish among them.

The system is called the Musician Hand. It was built by Hesam Azadjou, a Ph.D. candidate in the Valerolab.org of the Alfred E. Mann Department of Biomedical Engineering, under the direction of his adviser, Francisco Valero-Cuevas, professor of biomedical engineering, biokinesiology and physical therapy, mechanical engineering, electrical and computer engineering, and of computer science at USC.  Ali Marjaninejad, who completed his Ph.D. in biomedical engineering under the direction of Valero-Cuevas, contributed to the design of the methodology and formal analysis of the results.

Lead author Azadjou and co-authors Valero-Cuevas and Marjaninejad detail their findings in “Perception in Action: A Robotic System that Can Teach Itself to Melodiously Play Music by Ear,” a paper just published by the Royal Society Interface Journal.

For decades, robots have required exhaustive programming to function. Every action had to be spelled out in advance, every environment carefully controlled. This research suggests a different path: machines that learn from brief, real-world experience the way animals and humans do, then perform unscripted tasks in unpredictable environments like homes, hospitals and job sites.

Such an approach could one day make useful robots far cheaper and faster to deploy, and open the door to machines that learn and work alongside people in everyday life, not just on factory floors.

"The Achilles heel of traditional robotics is the assumption that perfect information is necessary to act well," said Valero-Cuevas. "Animals don’t work that way. They perceive, they guess, usually correctly, and they adapt. We wanted to show a robot could do the same."

Smarter Robots, Less Computing

Valero-Cuevas and Azadjou have done just that.

Most machines work only in tightly controlled environments where engineers prescribe and arrange every detail in advance. "You go to a factory, and the screws are in this bin and the nuts are in that bin," Valero-Cuevas said. "Everything is laid out so the robot knows exactly where to find things."

Step outside that controlled setting and conventional robots struggle. The environment changes. People are unpredictable. When acting in the real world, robots, such as self-driving cars,  compensate by requiring massive amounts of data, the equivalent of years of training, and heavy on-line computation.

"In robotics we tend to ask for more computing power, more data, more training time," Valero-Cuevas said. "What we are showing is that [robots] can learn useful behavior with very little data."

Azadjou, who trained originally as an electrical engineer, said the project challenged him to rethink his assumptions about how to solve a problem and to delve deeply into machine learning, biology, and neuroscience.

"Our brain solves incredibly complex problems using less than 100 watts of power, roughly the equivalent of an incandescent light bulb or your computer charger," he said. "To do the same thing with conventional AI, you might need megawatts. Nature is offering us a very different, energetically efficient kind of solution."

How the Robot Learned to Play-by-Ear

The Musician Hand has four fingers that press piano keys. Each finger is moved by a tendon, a thin cord connected to a small electric motor, much like how muscles pull tendons in a human hand.

To provide a suitable musical challenge, Valero-Cuevas brought this up with his piano teacher,  Richard Tuttobene—a Los Angeles-based professional composer/pianist/music theorist.  Tuttobene drew on his experience to compose `Robo Algo,’ an interesting four-note melody with musical dynamics which he played and recorded to present to the Musician Hand.

But before playing anything, the robot had to learn how its own “body” worked to produce sound by pressing on keys. The algorithm does not know it has a hand that is different from the keyboard, or what music is. It just experiences that its actions produce sounds.

The training process began with what the researchers call "motor babbling." For two minutes, the robot pressed keys in random patterns, varying force and timing. Each keystroke produced sound; the system recorded the relationship between finger actions and what it heard.

From that brief experience, the robot built an inverse map connecting the qualities of sounds to the qualities of motor commands needed to reproduce...