Robots Master Two-Handed Tasks Like Humans

Robots can now coordinate both arms to perform complex manipulation tasks with human-like efficiency, overcoming a longstanding challenge in robotics. Researchers at NVIDIA and UC San Diego have developed a system that enables robots to intelligently decide when to use their arms independently, sequentially, or collaboratively—just like humans do when tackling difficult two-handed jobs.

The key breakthrough is a planning system that allows robots to dynamically schedule which arm does what and when. Traditional robotic systems typically handle bimanual tasks by programming arms to act either completely independently or in strict sequence, leading to inefficient movements and frequent failures. This new approach treats bimanual coordination as an integrated scheduling problem, where the robot decides not just what actions to take, but how to best utilize both arms simultaneously.

The researchers built their system using a hierarchical framework that combines reinforcement learning with a Transformer-based scheduler. First, they trained individual robotic skills—like pushing, lifting, and pick-and-place operations—using GPU-accelerated simulation. These skills form a library of basic capabilities that both arms can access. Then, they trained a high-level scheduler that observes the current state and goal, and decides which skills to deploy from which arm, along with the specific parameters for each action.

The methodology involved creating a dataset of successful task demonstrations where the system had access to privileged information about optimal skill sequences. The Transformer-based policy learned to predict both which skill to use (from a set of five fundamental manipulation skills) and the precise parameters needed for execution. This approach balanced skill selection with parameter optimization through a combined loss function.

Experimental results demonstrated significant improvements over existing methods. In tasks requiring rearrangement of bulky objects into bins—scenarios where objects are too large for single-arm manipulation—the new system achieved a 51.3% success rate compared to just 5.5% for baseline reinforcement learning from scratch. It also showed 65.2% completion progress versus 26.4% for traditional methods, and reduced episode duration by 16% compared to sequential-only planning approaches.

The system's practical implications are substantial for real-world applications where robots need to handle complex manipulation tasks. In industrial settings, warehouse operations, or even domestic assistance, this technology could enable robots to efficiently perform tasks that currently require human dexterity—like moving large boxes, assembling furniture, or organizing spaces. The ability to dynamically switch between parallel, sequential, and collaborative arm movements makes robotic systems more adaptable to varying task requirements.

However, the research acknowledges limitations. The system was tested in simulated environments using the IsaacLab platform, and real-world validation remains necessary. The current skill library covers five fundamental manipulation types, but expanding this repertoire would be needed for more diverse applications. Additionally, the approach relies on pre-trained skills, meaning adaptation to entirely new manipulation types would require retraining those foundational capabilities.