UC San Diego Researchers Present TD-MPC2: Revolutionizing Model-Based Reinforcement Learning Across Diverse Domains

Large Language Models (LLMs) are constantly improvising, thanks to the advancements in Artificial Intelligence and Machine Learning. LLMs are making significant progress in sub-fields of AI, including Natural Language Processing, Natural Language Understanding, Natural Language Generation and Computer Vision. These models are trained on massive internet-scale datasets to develop generalist models that can handle a range of language and visual tasks. The availability of large datasets and well-thought-out architectures that can effectively scale with data and model size are credited for the growth.

LLMs have been successfully extended to robotics in recent times. However, a generalist embodied agent that learns to do many control tasks via low-level actions from a number of vast uncurated datasets still needs to be achieved. The current approaches to generalist embodied agents face two major obstacles, which are as follows.

1. Assumption of Near-Expert Trajectories: Due to the severe limitation of the amount of available data, many existing methods for behaviour cloning rely on near-expert trajectories. This implies that the agents are less flexible to different tasks since they require expert-like, high-quality demos to learn from.

2. Absence of Scalable Continuous Control Methods: Large, uncurated datasets cannot be effectively handled by a number of scalable continuous control methods. Many of the existing reinforcement learning (RL) algorithms rely on task-specific hyperparameters and are optimised for single-task learning.

As a solution to these challenges, a team of researchers has recently introduced TD-MPC2, an expansion of the TD-MPC (Trajectory Distribution Model Predictive Control) family of model-based RL algorithms. Big, uncurated datasets spanning several task domains, embodiments, and action spaces have been used to train TD-MPC2, a system for building generalist world models. It’s one of the significant features is that it does not require hyperparameter adjustment.

The main elements of TD-MPC2 are as follows.

1. Local Trajectory Optimisation in Latent Space: Without the need for a decoder, TD-MPC2 carries out local trajectory optimisation in the latent space of a trained implicit world model.

2. Algorithmic Robustness: By going over important design decisions again, the algorithm becomes more resilient.

3. Architecture for numerous Embodiments and Action Spaces: Without requiring prior domain expertise, the architecture is thoughtfully created to support datasets with multiple embodiments and action spaces.

The team has shared that upon evaluation, TD-MPC2 routinely performs better than model-based and model-free approaches that are currently in use for a variety of continuous control tasks. It works especially well in difficult subsets such as pick-and-place and locomotion tasks. The agent’s increased capabilities demonstrate scalability as model and data sizes grow. 

The team has summarised some notable characteristics of TD-MPC2, which are as follows.

1. Enhanced Performance: When used on a variety of RL tasks, TD-MPC2 provides enhancements over baseline algorithms.

2. Consistency with a Single Set of Hyperparameters: One of TD-MPC2’s key advantages is its capacity to produce impressive outcomes with a single set of hyperparameters reliably. This streamlines the tuning procedure and facilitates application to a range of jobs.

3. Scalability: Agent capabilities increase as both the model and data size grow. This scalability is essential for managing more complicated jobs and adjusting to various situations.

The team has trained a single agent with a substantial parameter count of 317 million to accomplish 80 tasks, demonstrating the scalability and efficacy of TD-MPC2. These tasks require several embodiments, i.e., physical forms of the agent and action spaces across multiple task domains. This demonstrates the versatility and strength of TD-MPC2 in addressing a broad range of difficulties.

Check out the Paper and Project. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 32k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, you will love our newsletter..

We are also on Telegram and WhatsApp.