AI Visualization Choices Skew Results

When artificial intelligence analyzes data, the way information is presented can significantly influence its conclusions—sometimes more than the actual data itself. A new study examining how multi-modal large language models (MLLMs) interpret network visualizations reveals that design choices can steer AI toward predetermined answers, regardless of underlying facts. This finding has critical implications for fields increasingly relying on AI for data analysis, from finance to healthcare.

The researchers discovered that including visualizations with network data strongly influenced AI decision-making, but not always in helpful ways. When asked to determine whether networks contained "bridges"—edges whose removal would disconnect the network—the AI's accuracy depended more on visualization style than on the actual network structure. In some cases, visualizations actually made the AI more likely to provide incorrect answers while maintaining high confidence in those wrong decisions.

The study tested two AI models, GPT-4 and Qwen, using three common visualization styles: pixel-based adjacency matrices, circular layouts, and spring (force-directed) layouts. Researchers presented the AI with 200 different network configurations, asking whether each contained a bridge. They compared performance when the AI received only text-based network data versus when it also received visualizations.

Results showed stark differences in how visualization styles affected decision-making. Pixel-based adjacency matrices made both AI models more likely to report that bridges existed, achieving high accuracy for networks that actually contained bridges (up to 94% for GPT-4) but poor performance for networks without bridges (as low as 6% accuracy for Qwen). Spring layouts had the opposite effect, making AI more likely to deny bridge existence. Circular layouts fell somewhere in between, though they consistently reduced AI confidence compared to text-only inputs.

Perhaps most concerning was the disconnect between accuracy and confidence. Both AI models reported high confidence scores (typically 4-5 on a 5-point scale) regardless of whether their answers were correct. When visualizations were included, this confidence remained high even when accuracy dropped significantly. The structured text-based representation actually resulted in lower confidence scores than the unstructured version, despite containing the same information.

These findings matter because AI is increasingly used for critical analysis tasks where visualization choices could inadvertently steer conclusions. In applications like financial risk assessment, medical diagnosis, or legal analysis, biased AI interpretations could have real-world consequences. The study suggests that simply adding visualizations to AI inputs doesn't necessarily improve decision quality—it may instead introduce new sources of error.

The research has several limitations. It focused on a single task (bridge detection) using specific visualization algorithms. Other tasks or more advanced visualization techniques might yield different results. The study also used zero-shot prompting without allowing the AI to access external tools, which might affect performance. Future work could explore whether different prompting strategies or model architectures reduce this visualization bias.