AI Learns to Prune Vision Systems Like a Gardener

Vision Transformers, the powerful AI models behind many image recognition systems, have a hidden inefficiency: they often use more computational resources than necessary. While dynamic head pruning techniques can remove redundant attention heads to improve efficiency, these s have traditionally been opaque and difficult to control. Researchers from the Korea Advanced Institute of Science and Technology have developed a new approach that makes this pruning process both interpretable and controllable, potentially leading to more efficient and understandable AI vision systems.

The key finding is that by using Sparse Autoencoders (SAEs) to analyze and manipulate the internal representations of Vision Transformers, researchers can steer pruning decisions in class-specific ways. This approach reveals that different object categories rely on distinct subsets of attention heads within the model. For example, when recognizing bowls, the system learned to rely primarily on just two specific heads (h2 and h5) while maintaining high accuracy. This selective pruning reduced head usage from 0.72 to 0.33 while actually improving accuracy from 76% to 82% for bowl recognition. Similarly, pine tree recognition showed improved accuracy (79% to 84%) with reduced head usage (0.93 to 0.35) through reliance on heads h2 and h3.

Ology integrates Sparse Autoencoders with the AdaViT dynamic pruning framework. The researchers first trained a Vision Transformer on CIFAR-100 using ImageNet-pretrained ViT-Small architecture with 12 layers and 6 heads per layer. They then extracted the CLS token from the final layer's residual input and trained a k-sparse autoencoder to expand the 384-dimensional embedding into a 3072-dimensional latent space. This SAE was trained for 100 epochs, achieving a mean squared error loss of 0.0228. The researchers then amplified selected latent dimensions using different strategies—per-class frequent activations, global frequent activations, and random selection—to observe how these manipulations affected pruning decisions when fed back into the decision network.

, Detailed in Figures 2 and 3 of the paper, show that per-class steering was particularly effective. While global and random strategies led to significant accuracy drops as amplification strength increased, per-class steering maintained accuracy while reducing head usage. The low overlap between global and per-class top-k frequent latent dimensions (0.1641) indicates that the SAE captures class-discriminative concepts. Additional analysis in the appendix reveals that semantically related classes like bowl and plate share similar head subsets (h2 and h5), while unrelated classes show near-zero latent overlap. Negative steering experiments (α < 0) demonstrated that suppressing class-specific latent features increased head usage, confirming that these features directly influence pruning decisions.

This research matters because it addresses two critical s in modern AI: efficiency and interpretability. By making pruning decisions controllable at the latent level, this approach could lead to more efficient vision systems that use computational resources more selectively. The ability to understand which heads are important for specific object categories provides valuable insights into how these complex models make decisions. This could have practical for deploying AI vision systems in resource-constrained environments like mobile devices or edge computing scenarios, where computational efficiency is paramount.

The study acknowledges several limitations that future work must address. The current framework focuses only on the final layer of the Vision Transformer and was tested on relatively small datasets like CIFAR-100. The researchers note that extending this approach to earlier layers and foundation models represents an important direction for future research. Additionally, while demonstrates class-specific control, its scalability to larger, more complex datasets and its generalization to different model architectures remain open questions that require further investigation.