This robot hand can manipulate objects without seeing them

Listen to this article

Researchers from Columbia Engineering have designed a robotic hand that joins an advanced sense of touch with motor-learning algorithms, resulting in a robotic hand that doesn’t rely on vision to manipulate objects.

The robotic hand, equipped with five fingers and 15 independently actuated joints, is able to execute an arbitrarily large rotation of an unevenly shaped grasped object in hand while also maintaining the object in a stable, secure hold without any visual feedback whatsoever. This is a difficult manipulation task for robots because it requires constant repositioning of a subset of fingers, while other fingers must keep the object stable. 

“While our demonstration was on a proof-of-concept task, meant to illustrate the capabilities of the hand, we believe that this level of dexterity will open up entirely new applications for robotic manipulation in the real world,” Matei Ciocarlie, associate professor in the Departments of Mechanical Engineering and Computer Science, said. “Some of the more immediate uses might be in logistics and material handling, helping ease up supply chain problems like the ones that have plagued our economy in recent years, and in advanced manufacturing and assembly in factories.”

While robots out in the world are performing more and more complex manipulation tasks, the vast majority, if not all of them, need vision capabilities to do so. Columbia University’s robot hand is immune to lighting issues, occlusion, or other similar issues that could prevent a robot that does rely on vision capabilities from working properly. 

For this research, Ciocarlie’s team built off of previous work they did with Ioannis Kymissis, a professor of electrical engineering, where they built a new generation of optics-based tactile robot fingers. These robotic fingers were able to achieve contact localization with sub-millimeter precision while also providing complete coverage of a complex, multi-curved surface. 

Each finger on this new robotic hand is equipped with the team’s touch-sensing technology. The team, led by Gagan Khandate, Ciocarlie’s doctoral researcher, tested its ability to perform complex manipulation tasks using new methods for motor learning.

In particular, the team utilized a method called deep reinforcement learning, and augmented it with new algorithms that they developed for effective exploration of possible motor strategies. These motor learning algorithms used exclusively tactile and proprioceptive data to learn. Using these methods, the robot completed about a year of practice in only hours of real-time.  

The Columbia Engineering team then transferred the manipulation skills learned in simulation to the real robot hand, which could then achieve a high level of dexterity. 

“In this study, we’ve shown that robot hands can also be highly dexterous based on touch sensing alone. Once we also add visual feedback into the mix along with touch, we hope to be able to achieve even more dexterity, and one day start approaching the replication of the human hand,” Ciocarlie said.