Generative AI and Isaac to power smarter robots, says NVIDIA before CES

Listen to this article

Generative AI can help build warehouse scenes such as this one created in NVIDIA Omniverse and Isaac Sim. Credit: NVIDIA

Last year, generative AI received a lot of attention, but the challenge in 2024 is to apply it to industrial technologies and commercial use cases. NVIDIA Corp. and its partners are working with generative AI to make robots more capable, said Deepu Talla, vice president of robotics and edge computing at NVIDIA.

The company’s graphics processing unit (GPU) can accelerate large language models (LLMs) to improve machine adaptability, he said in a special address before NVIDIA’s presentations at CES in Las Vegas this week.

“Autonomous robots powered by artificial intelligence are being increasingly utilized for improving efficiency, decreasing costs, and tackling labor shortages,” according to Talla.

NVIDIA provides foundation for generative AI

A decade ago, Jensen Huang, founder and CEO of NVIDIA, delivered the first NVIDIA DGX supercomputer to OpenAI, whose ChatGPT is a now leading example of generative AI, noted the company.

Such technology “will go beyond text and image generation — and into homes and offices, farms and factories, hospitals and laboratories,” asserted Talla. LLMs will enable robots to more easily understand human instructions, learn from one another, and comprehend their environments, he said.

NVIDIA said its Isaac and Jetson platforms facilitate the development and deployment of AI-powered robots. More than 1.2 million developers and 10,000 partners and customers already rely on the platforms.

More than 20 of them are at CES this week, said Gerard Andrews, senior product marketing manager for robotics at NVIDIA. They include:

• Analog Devices
• Aurora Labs
• Canonical
• Dreame Innovation Technology
• DriveU
• e-con Systems
• Ecotron
• Enchanted Tools
• GlüxKind
• Hesai Technology
• Leopard Imaging
• Segway-Ninebot (Willand Beijing Technology Co.)
• Nodar
• Orbbec
• QT Group
• Robosense
• Spartan Radar
• TDK Corp.
• Telit
• Unitree Robotics
• Voyant Photonics
• ZVISION Technologies Co.

A two-brain model for AI in robotics

Talla described a dual-computer model for robots using AI. The first computer, or “AI factory,” creates and continually improves AI models with NVIDIA’s compute infrastructure and NVIDIA Omniverse for simulation and training.

The second computer represents the runtime environment of the robot, which can vary by application, explained Talla. It could be in the cloud or a data center, in an on-premises server for tasks such as defect inspection in semiconductor manufacturing, or within an autonomous machine equipped with multiple sensors and cameras, he said.

For example, Electric Sheep is using Isaac Sim in its AI factory to build a foundational world model for outdoor autonomy to train mowing robots, said Andrews.

He also cited other ecosystem partners that are using LLMs with legged robots, manipulators, and agricultural robots. They included Boston Dynamics using Spot and ChatGPT as a tour guide (see video below), Collaborative Robotics using an LLM “front end” to translate prompts into commands for mobile applications, and Covariant building a foundational model for automating warehouses with tens of thousands of SKUs.

In addition, Andrews mentioned Agility Robotics, NTT, and Sanctuary AI. “Generative AI makes robotics smarter and more capable,” he said.

NVIDIA ecosystem partners are using GPU-accelerated LLMs and generative AI to make robots more adaptable. Click here to enlarge. Credit: NVIDIA

LLMs add creativity, realism to robot training

NVIDIA said generative AI tools such as NVIDIA Picasso allow users to generate realistic 3D assets from text prompts and add them to digital scenes. The company stated that more realistic environments, along with the physics-accurate scenarios in Omniverse, make training and testing of robots in simulation more applicable to real-world operations.

“Generative AI models can improve robotics perception and planning for a better understanding of their environments and the ability to execute skills,” said Andrews.

Since most robots are designed for specific tasks, modifying them to do new ones can be time-consuming, said Talla. Advances in LLMs and vision language models will overcome this bottleneck and enable more intuitive human-machine interactions, he said.

Simulation could also help collaborative robots with balancing safety and responsiveness, said Andrews. He said he expects generative AI and simulation to become layers in the software stack that generates plans and implements safety policies.

NVIDIA made several other announcements around CES, including how its cloud GPUs, simulation software, and partnerships will enhance LLM performance for digital avatars, image generation, video games, and livestreaming. Automakers will soon go into production with new electric vehicles developed with NVIDIA Drive, said the company.

Editor’s note: Andrews and WTWH Media robotics editor Mike Oitzman will be participating in a CES panel on “Robots at Work” at N258 in the Las Vegas Convention Center at 3:00 p.m. PT on Tuesday, Jan. 9.