Stanford and MIT CSAIL Researchers Propose ‘RoboCraft’: A Novel Framework That Allows Robots To Manipulate Deformable Materials From Visual Inputs Using Graph Neural Networks (GNNs)

To perform difficult industrial and domestic tasks, robots must be able to model and manipulate elastoplastic items (e.g., stuffing dumplings, rolling sushi, and making pottery). Although highly plastic materials like dough and plasticine are frequently used in residential and industrial contexts, manipulating them presents a special set of difficulties for robots. 

Soft and deformable objects have a high degree of freedom (DoF), incomplete observability, and nonlinear interactions between local particles. These characteristics make controlling deformable objects difficult at nearly every step of the robotic manipulation pipeline, including describing states, simulating dynamics, and creating control signals.

A recent study by researchers at Stanford University and MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) explores ways to enable robots to model and operate elastoplastic items based on unprocessed RGBD visual data. In their paper “RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks,” they demonstrate that the proposed “RoboCraft” model could accurately predict how a robot would behave when pinching and releasing Play-Doh to form different letters, even ones it had never seen before. The two-finger gripper performed on par with, occasionally even better than, human counterparts who teleoperated the system with just ten minutes of data.

Before performing any type of efficient and successful modeling or planning on an undefined, smooth material, the entire structure must be considered. Furthermore, altering one part of a flexible structure affects other parts too.

Some earlier techniques directly relied on learning dynamics models from high-dimensional sensory data. In contrast, others used particles to represent deformable objects and graph neural networks (GNNs) to simulate their dynamics. These models, however, fall short since the structure of the items is not explicitly exploited. Raw sensory data cannot provide such robust monitoring, which further restricts their utility in practical applications.

RoboCraft, employing a graph neural network as the dynamics model, is able to more accurately forecast how the material will change shape by converting the photos into graphs of tiny particles and coupling them with algorithms. RoboCraft employs visual data instead of complicated physics simulators, which researchers have traditionally utilized to simulate and comprehend force and dynamics being applied to objects.

The new framework is composed of three main components: 

1. A perception module that builds a particle representation of the object by sampling from the mesh of the reconstructed object
2. A dynamics model that employs GNNs to model particle interactions. The dynamic model is trained directly from raw visual data using loss functions that assess the difference between anticipated and observed particle distributions, in contrast to past learning-based particle dynamics works that presume temporal correlation.
3. A planning module that applies model-predictive control (MPC) to solve the trajectory optimization problem. We train the dynamics model. 

The team is now working on creating dumplings out of dough and a pre-made filling in addition to making adorable forms. 

Overall, RoboCraft shows that predictive models can be taught to plan motion in very data-efficient ways. The team believes that their system can be used in manipulating materials with a variety of instruments and will be able to help with home chores and duties, which may be especially beneficial for the elderly or those with restricted mobility. In the future, they plan to help the model comprehend and complete longer-term planning tasks, such as predicting how the dough will deform given the current tool, movements, and actions.

his Article is written as a summary article by Marktechpost Staff based on the paper 'RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks'. All Credit For This Research Goes To Researchers on This Project. Checkout the paper and blog post.

Please Don't Forget To Join Our ML Subreddit