Robots Learn Complex Tasks by Choosing What to Practice

Autonomous robots that can learn and adapt in complex environments are essential for applications from manufacturing to exploration, but teaching them to handle multiple, interconnected tasks without constant human input remains a major challenge. In a study published in arXiv, researchers present a novel approach that allows robots to decide which skills to practice next, enabling them to efficiently master tasks that depend on one another. This advancement could lead to more versatile and independent robotic systems in real-world settings.

The key finding is that treating task selection as a Markov Decision Process (MDP)—where the robot considers how current choices affect future learning—significantly improves performance in scenarios with interdependent tasks. In experiments, the system, called Markovian-GRAIL (M-GRAIL), achieved high competence levels (close to 90% success rates) across sequences of tasks, such as activating spheres in a chain where each task is a precondition for the next. This contrasts with simpler methods that struggle when tasks are interrelated, as they fail to prioritize learning in a way that accounts for these dependencies.

Methodologically, the researchers built on an existing architecture called GRAIL, which uses intrinsic motivation signals to drive learning. They modified the goal-selection component to incorporate environmental context and temporal dependencies. In the M-GRAIL system, goal selection is modeled as an MDP and solved with Q-learning, a reinforcement learning algorithm that assigns values to goals based on long-term benefits. This allows the robot to propagate motivation from later tasks back to their prerequisites, ensuring it practices foundational skills first. The setup involved a simulated iCub robot with two arms, where tasks included touching spheres to activate them, with some tasks requiring others to be completed first.

Results from three experiments highlight the effectiveness of this approach. In the first experiment, with no task relations, all systems performed well, but in the second, where tasks depended on environmental conditions, the contextual version (C-GRAIL) outperformed the baseline by adapting to context changes. In the third and most complex experiment, with chains of interdependent tasks, M-GRAIL achieved near-complete learning of all tasks, while other systems wasted trials on unachievable goals and failed to master later tasks in the chains. For instance, M-GRAIL reduced wasted trials to nearly zero, as shown in Figure 8, by systematically selecting goals that build on previous achievements.

This work matters because it addresses a core issue in robotics: enabling lifelong, open-ended learning where robots can acquire skills autonomously. In practical terms, such systems could be deployed in dynamic environments like warehouses or disaster zones, where tasks are interconnected and pre-defined instructions are impractical. By using intrinsic motivations—self-generated signals that guide exploration—the approach avoids the need for external rewards, making it suitable for scenarios where goals are not pre-specified.

Limitations noted in the paper include that the system does not retain learned task chains as hierarchical skills after training, meaning it cannot reuse sequences without relearning. Additionally, the study used simulated environments and focused on specific tasks, so generalizing to real-world robots or different controllers remains an area for future research. The authors suggest that integrating planners or visual guidance could address these constraints, but for now, the method excels in structured, interdependent scenarios without external oversight.