Shanghai Jiao Tong University Researchers Unveil RH20T: The Ultimate Robotic Dataset Boasting 110K Sequences, Multimodal Data, and 147 Diverse Tasks

Robotic manipulation is advancing towards the goal of enabling robots to swiftly acquire new skills through one-shot imitation learning and foundational models. While the field has made strides in simple tasks like object manipulation, hurdles impede progress in more complex scenarios. The scarcity of large and diverse robotic manipulation datasets and a reliance on visual guidance are key challenges. To address these issues, researchers from Shanghai Jiao Tong University introduce an innovative data collection approach employing force-torque sensors and haptic devices.

There are three critical areas in robotic manipulation research: the scarcity of comprehensive datasets, the promising advancements in one-shot imitation learning and foundational models, and the necessity of integrating visual and tactile perception for complex skill acquisition. The researchers recognize the untapped potential within one-shot learning and foundational models to elevate robotic manipulative skills by harnessing the power of demonstrations.

Researchers tackle the challenge of equipping robots with diverse and adaptable skills for open-domain tasks using one-shot imitation learning and foundational robotic models. While current efforts primarily revolve around straightforward tasks like pushing or picking objects, mainly guided by visual cues, the potential for more complex skills involving both visual and tactile perception remains unexplored. Their approach introduces an innovative data collection approach for robotic manipulation, integrating a force-torque sensor and a haptic device to gather data. Their dataset comprises over 110,000 robot manipulation sequences spanning various skills, scenarios, robots, and camera angles, encompassing visual, force, audio, and action data.

The importance of intuitive teleoperation, their research highlights its role in collision avoidance and generating significant forces safely. Their organized dataset, designed to be representative, diverse, and true to real-world scenarios, promises to be a valuable asset for advancing research in general skill learning. The primary focus lies in demonstrating how their dataset enhances the transferability of a baseline model within a few-shot learning framework.

Their research showcases the model’s performance across various training configurations, highlighting the substantial benefits of leveraging the diverse dataset for robotic manipulation. Pretraining the model with dataset data, despite differing conditions, significantly boosts success rates. The incorporation of data from diverse tasks during pre-training further enhances overall performance and accelerates model convergence. Notably, the dataset proves its value in few-shot learning, with pretrained models consistently outperforming their non-pre-trained counterparts, even with fewer demonstrations. Their research substantially bolsters the model’s generalization capabilities, consistently outshining non-pretrained models when tested in new environments.

In conclusion, their dataset provides a valuable resource for diverse robotic skill learning, particularly in the field of robotic manipulation in novel environments. It provides contact-rich robot manipulation sequences across various skills, contexts, robots, and camera viewpoints, with multimodal perception information. While acknowledging limitations, like the high data collection costs and the need for further evaluation with robotic foundation models, the researchers have generously open-sourced the dataset to foster collaboration and progress in the field. Future endeavors aim to expand the dataset to encompass a wider range of robotic manipulation tasks, including dual-arm and multi-finger dexterous manipulation.

Check out the Paper and Project. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 31k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, you will love our newsletter..