AI and Humans Team Up for Better Decisions

In complex, fast-paced situations like military coalitions or emergency responses, teams often struggle to integrate human expertise with artificial intelligence effectively. Researchers have developed a method called human-agent knowledge fusion (HAKF) to address this, enabling smoother collaboration between people and AI agents. This approach is crucial for improving decision-making in unpredictable settings where pre-built solutions fall short.

The key finding is that HAKF enhances both human trust in AI and AI performance through two main communication flows: tellability and explainability. Tellability allows humans to share local knowledge with AI systems, such as marking certain activity classifications as 'regular' to ignore them in analyses. Explainability provides transparency into AI decisions, like highlighting which parts of a video or audio feed influenced a classification, helping users understand and trust the system's outputs.

Methodologically, the researchers used a graphical environment called Cogni-Sketch to implement HAKF. This tool represents information as a knowledge graph, where concepts and relationships are defined using simple semantics based on first-order predicate logic. Human and machine agents interact through visual palettes, dragging and dropping elements to configure services. For example, in the activity recognition capability, pre-trained models classify activities from data feeds, and users can map these generic outputs to domain-specific concepts. The environment supports neuro-symbolic processing, combining rule-based systems with neural networks, and is accessible via various interfaces, including conversational and visual modes.

Results from the worked examples show practical applications. In one scenario, users applied the selective audio-visual relevance (SAVR) explainability technique to a nightclub camera feed, identifying frequent classifications like 'shotput' and 'hammer throw' and marking them as regular to filter out noise. This customization improved the system's relevance to the specific operation. In another example, users defined complex events, such as an improvised explosive device (IED) event, by combining simple audio events like explosions and sirens with spatio-temporal constraints. The system generated ProbLog code to detect these events, enabling rapid deployment in coalition settings. The data, illustrated in Figures 1, 2, and 3 from the paper, demonstrates how these interactions increase confidence and performance without requiring retraining of AI models.

Contextually, this matters because it allows non-experts to quickly adapt AI tools to real-world scenarios, such as monitoring multiple data feeds for threats in coalition operations. By making AI more transparent and customizable, HAKF reduces the cognitive burden on users and supports agile responses in high-tempo environments. This could benefit fields like disaster response or security, where timely, informed decisions are critical.

Limitations include the need for further evaluation, as plans for testing with large user groups and subject matter experts are still under development. The paper notes that future work will explore better ways to convey uncertainty from multiple sources and integrate autonomous negotiation agents, but current implementations may not handle all dimensions of uncertainty effectively in dynamic settings.