Robots Follow Natural Language Commands Without Special Training

Robots that can understand and execute complex natural language commands without specialized training represent a significant step toward more adaptable and accessible robotic systems. This research demonstrates how combining existing artificial intelligence models can create robots that follow instructions ranging from simple object relocation to puzzle solving, potentially lowering barriers to robotic deployment in various settings.

The key finding is that robots can successfully perform long-horizon manipulation tasks by integrating multiple foundation models without requiring domain-specific training. The framework translates verbal commands into executable action sequences, maintaining accuracy while using pre-trained models off-the-shelf. This approach avoids the massive training datasets typically needed for vision-language-action models while achieving robust performance across diverse tasks.

Methodologically, the researchers developed a layered architecture that combines different AI models for distinct functions. The system uses a large language model for natural language interpretation and planning, a vision-language model for perception and spatial reasoning, and maintains a scene graph that encodes object relationships and properties in the environment. This modular design allows each component to specialize while working together through the Robot Operating System 2 framework. The hardware setup employed a UR10e robotic arm with a 6-degree-of-freedom reach and a wrist-mounted 3D camera for environmental perception.

Results from tabletop experiments show the framework achieved high performance across multiple task categories. In fundamental capability tests, planning feasibility reached 100% for tasks like moving objects between specified positions and semantic clustering. Task completion rates were 100% for well-specified instructions, though they dropped to 20% for ambiguous commands lacking sufficient context. The system successfully solved the Tower of Hanoi puzzle, demonstrating constraint-aware sequencing, and achieved 75% completion in table organization tasks requiring multi-step reasoning. Scene graph handling maintained perfect accuracy across all experiments, showing the system's ability to consistently track environmental changes.

This work matters because it provides a practical middle ground between data-intensive approaches that require massive training and purely symbolic planners that lack physical grounding. The ability to use off-the-shelf models reduces engineering burden and could make sophisticated robotic manipulation more accessible to researchers and applications where collecting specialized training data is impractical. The framework's generalization to novel object arrangements without retraining suggests potential for real-world deployment in environments with unpredictable configurations.

Limitations include sensitivity to ambiguous language instructions, where sparse user requests sometimes caused the system to issue insufficient queries. Error propagation between layers occasionally occurred, particularly in perception and localization. The system does not model collisions between manipulatable objects, reducing task completion rates in cluttered scenes. These constraints highlight areas where further development is needed for robust real-world application.