Robots Learn Complex Tasks by Watching Humans

In hazardous environments like offshore oil platforms, robots must perform complex inspection tasks reliably alongside human operators. Traditional programming methods often fall short when rapid adaptation is needed. Researchers from the University of Edinburgh have developed a system that allows robots to learn from human demonstrations and generate their own interpretable programs, enhancing both autonomy and safety in critical industrial settings.

The key finding is that robots can now automatically infer high-level goals from human demonstrations and synthesize computer programs that represent these tasks. This approach enables robots to perform surveillance and inspection duties in environments they haven't encountered before, while providing explanations for their behavior. The system successfully learned hybrid controllers for an unmanned ground vehicle to inspect specific areas of an oil rig digital twin in a particular order, as shown in the experimental results.

The methodology combines semi-supervised learning with program synthesis. The system first records human operators performing tasks through teleoperation, capturing sensory inputs and actions. Using sequential importance sampling with attribution priors, it identifies proportional controllers that guide the robot toward goals while avoiding obstacles. The system then clusters these controllers to discover the state space and automatically labels demonstration segments. This allows the induction of computer programs that abstract the original demonstration into interpretable code, as illustrated in Listing 1 of the paper.

Results show the system successfully inferred transit goals and avoidance behaviors in various scenarios. In Figure 2, the approach identified controllers that accounted for obstacles, unlike baseline methods that failed when obstacles were present. The system demonstrated the ability to run synthesized programs in unseen environmental configurations while maintaining performance. The induced programs, such as those shown in Listings 3-5, represent complex surveillance patterns including loops and conditional flows that weren't present in the original demonstrations.

This advancement matters because it addresses the critical need for robots that can work autonomously in dangerous industrial environments while remaining interpretable to human operators. Unlike black-box AI systems, this approach produces programs that operators can examine directly, understanding what the robot will do before execution. The system's ability to provide causal analysis and counterfactual explanations, as demonstrated in Figures 7 and 8, helps operators diagnose failures and modify robot behavior without physical intervention.

Limitations include the requirement that robot behavior must be satisfiable by proportional controllers, restricting application to certain types of tasks. The system also assumes parts of the environment model are known, and switching between subsequent controllers only occurs when the previous controller has finished its task. These constraints mean the approach may not generalize to all robotic applications without modification.