AI Reveals How Vision-Language Models Think

As artificial intelligence systems like CLIP become more powerful and widely used, understanding their inner workings has grown increasingly urgent. These vision-language models, which connect images and text, often operate as black boxes, making it hard to know why they make certain decisions or how they represent concepts. This opacity raises concerns about biases, safety, and reliability in applications ranging from content moderation to medical diagnostics. A breakthrough from researchers at the University of Trento, University of Pisa, and Fondazione Bruno Kessler offers a novel solution: a that peeks inside these models without needing any external data, revealing their semantic building blocks in a human-interpretable way.

The key finding is that the attention heads in CLIP's vision transformer encode distinct, coherent concepts directly in their weights, which can be extracted and manipulated. The researchers discovered that by decomposing the weight matrices of these attention heads using Singular Value Decomposition (SVD), they could isolate individual semantic directions—such as colors, textures, or locations—within each head. For example, one singular vector might specialize in the color green, while another focuses on outdoor scenes like beaches. This granular insight allows for precise interventions, such as suppressing unwanted concepts like nudity or amplifying task-relevant features, all without altering the model's training or requiring additional datasets. The approach, called SITH (Semantic Inspection of Transformer Heads), moves beyond previous s that relied on activations and were prone to dataset biases, offering a more stable and interpretable view of the model's knowledge.

To achieve this, the team developed a two-step ology that is entirely data-free and training-free. First, they isolated the value-output (VO) weight matrix from each attention head in CLIP's vision transformer, which governs how information is written into the model's internal stream. They then applied SVD to this matrix, breaking it down into singular vectors that represent the dominant computational directions. Each singular vector was interpreted using a new algorithm called COMP (Coherent Orthogonal Matching Pursuit), which maps these vectors to sparse combinations of human-understandable concepts from a large pool like ConceptNet. COMP optimizes for both reconstruction fidelity—ensuring the concepts accurately capture the vector's meaning—and semantic coherence, so the explanations are clear and meaningful. This process avoids the need for large image datasets, eliminating biases that can skew interpretations in activation-based s.

Demonstrate that SITH provides faithful and interpretable explanations, as validated through quantitative experiments. In Figure 3 of the paper, the researchers show that COMP achieves a balance between interpretability and fidelity, outperforming baselines like top-k selection and non-negative orthogonal matching pursuit. For instance, with a sparsity level of 5 concepts per vector, COMP achieved interpretability scores around 4.0 and fidelity scores near 0.6, indicating high coherence and accurate reconstruction. Table 2 lists examples of singular vectors and their interpreted concepts, such as "pink red" and "red telephone" for a vector in layer 23, head 8. Moreover, image matching experiments in Figure 5 confirm that the attributed concepts align with visual evidence, with top-retrieved images for a vector interpreted as "women in sports" showing athletic scenes. also enabled effective model edits: suppressing spurious background features improved worst-group accuracy on the Waterbirds dataset from 47.9% to 70.6%, as shown in Table 3.

This breakthrough has significant real-world for making AI systems more transparent, safe, and adaptable. By enabling data-free interpretability, SITH allows developers to audit models for biases or unsafe content without exposing sensitive data. For example, it can identify and suppress concepts related to violence or nudity, enhancing safety in content retrieval tasks—Table 4 shows improved performance on the ViSU dataset for unsafe queries. Additionally, supports task-specific enhancements: amplifying relevant singular vectors boosted zero-shot classification accuracy on datasets like Flowers 102 by up to 1.0 percentage points, as seen in Table 5. Beyond editing, SITH sheds light on how models adapt during fine-tuning, revealing that changes primarily reweight existing semantic bases rather than learning entirely new features, which can inform more efficient training strategies. This could lead to more trustworthy AI in fields like healthcare, where understanding model decisions is critical.

Despite its advantages, SITH has limitations that point to future research directions. The paper notes that not all singular vectors can be perfectly reconstructed, as some may encode non-semantic or noisy information that doesn't align with human concepts. This is evident in Figure 3, where even with 50 concepts per vector, fidelity scores plateau, suggesting inherent limits in interpretability. Additionally, currently focuses on the vision transformer's attention heads; extending it to other components like feed-forward networks or query-key matrices could provide a more comprehensive view. The researchers also highlight that the concept pool, while large, may not cover all possible semantic nuances, and the approach assumes a stable weight structure that might vary in dynamically trained models. These s underscore the need for continued work to fully demystify complex AI systems.