Atlas-Alignment: Making Interpretability Transferable Across Language Models

Artificial intelligence models are increasingly used in critical areas like healthcare and finance, yet their inner workings remain a black box, raising concerns about safety and reliability. A new study introduces a technique that could make these systems transparent and steerable, addressing a major barrier to trust in AI.

The key finding is that different AI models encode human-understandable concepts in similar ways, allowing one interpretable model to serve as a universal reference—called a Concept Atlas. By aligning an opaque 'subject model' with this atlas using lightweight mathematical transformations, researchers can identify and control specific concepts within the model without needing extensive labeled data or retraining. For example, they demonstrated steering a model's output toward themes like 'secrets and deception' by modifying internal activations.

The methodology relies on aligning the latent spaces of models through techniques like Orthogonal Procrustes, which acts as a simple rotation or reflection to map one model's representations to another. This process uses only input data—feeding the same text sequences to both the subject model and the atlas model—and compares their activations to learn the mapping. It avoids the need for manual labeling or training new components from scratch, building on existing sparse autoencoders that decompose complex features into interpretable units.

Results from quantitative evaluations show the method's effectiveness. In feature identification, Orthogonal Procrustes achieved an average precision of 0.40–0.49, significantly outperforming baselines like covariance methods, which scored around 0.046. For semantic retrieval, it reached near-perfect scores in metrics like Mean Reciprocal Rank (up to 0.94), indicating it reliably recovers correct concepts. In steering experiments, this method increased concept expression in model outputs by over 30% in some cases, compared to minimal effects from other approaches, as measured by automated ratings of generated text.

This work matters because it amortizes the cost of interpretability—investing once in a high-quality atlas allows many models to become transparent with minimal additional effort. It enables practical applications, such as ensuring AI systems avoid biased or harmful behaviors by allowing users to inspect and steer their reasoning processes. For instance, in content generation, it could help guide models to produce more accurate or ethical outputs, enhancing reliability in real-world deployments.

Limitations include the assumption that models learn comparable concepts, which may not hold perfectly across all architectures or domains. The current approach also discards positional information through max-pooling, potentially missing nuances in sequential data. Future work could address these issues to improve robustness and expand the framework to other model components like attention heads.