Inside Meta’s research with Boston Dynamics’ Spot

Listen to this article

Researchers at Meta AI and Fundamental AI Research (FAIR) have been working with Boston Dynamics‘ Spot quadruped to push the robot to new heights. Their research resulted in two significant breakthroughs toward creating general-purposed embodied AI agents that are capable of performing challenging sensorimotor skills. 

While Spot has been at work with industrial users since 2019, one of its primary purposes, according to the Boston Dynamics team, is for researchers who want to use the robot as a platform to push the field forward. 

When they started working with Spot, Meta researchers were particularly interested in making Spot better at high-level reasoning and planning, making it able to handle unfamiliar environments and understand simple, natural language instructions.  

Meta and FAIR’s team trained three Spot robots with simulation data. This training involved allowing the robots to see what it looks like to retrieve everyday objects in various settings, including home, apartment, and office settings. The team then tested the robots’ ability to navigate new spaces and overcome unexpected obstacles to retrieve those objects in the real world working across three different locations in California, New York, and Georgia. 

“Compared with a more traditional way of doing the same tasks, we found that we could get much higher success because our policies were more robust, and they allowed the robot to deal with disturbances that happened in the real world,” Akshara Rai, a research scientist on the FAIR team, said. “If the object is not where it is supposed to be, the robot can re-plan based on the environment and the information that the robot has. Spot is already very good at navigating an environment if we give it a map beforehand. The most important thing we’re adding is this generalization to a completely unseen environment.”

With these methods, the team was able to develop an artificial visual cortex called VC-1, the team’s first breakthrough. VC-1 matches or outperforms best-known results on 17 different sensorimotor tasks in virtual environments. 

The second breakthrough the team made was developing a new approach called adaptive, or sensorimotor, sill coordination (ASC). ASC achieves near-perfect performance on robotic mobile manipulation testing. With ASC, Spot succeeded in 98% of its attempts to locate and retrieve an unfamiliar object, compared to just a 73% success rate with traditional methods. 

“The way that Meta is using Spot is exactly how we hoped people would use the robot when we designed it,” Zack Jackowski, general manager for Spot at Boston Dynamics, said. “Right now, Spot can walk a repeatable path through an industrial facility and keep track of equipment performance, and that’s valuable. We would all love it if we could get to the point where we can say, ‘Hey Spot, go take a look at that pump on the floor there.’ That’s the kind of thing that the Meta team is working on.”