AI's Multimodal Leap Relies on Rare Breakthroughs, Not Gradual Change

A new analysis of over 1.8 million AI models shows that the ability to process both images and text—a key feature in modern artificial intelligence—does not spread gradually through incremental updates. Instead, it emerges in sudden bursts when rare 'founder' models are introduced, after which these multimodal capabilities rapidly expand within their own lineages. This pattern, drawn from the ModelBiome AI Ecosystem dataset of Hugging Face models, explains why major open-source language model families lagged behind the broader ecosystem in adopting vision-language tasks until 2024-2025, despite earlier availability. suggest that multimodal innovation in AI is bottlenecked by integration events that require substantial engineering effort, rather than being a simple extension of text-only development.

Researchers quantified the timing of multimodality by examining task tags and lineage records in the dataset, which includes 1.86 million model entries and 3.02 million parent-child relationships. They found that cross-modal tasks, especially image-text vision-language models (VLMs), were common in the broader Hugging Face ecosystem well before they became prevalent within major open LLM families. Within these families, multimodality remained rare through 2023 and most of 2024, then increased sharply in 2024-2025, dominated by image-text tasks. For example, the first VLM variants appeared months after the initial text-generation releases in families like Gemma (with a lag of about one month) and GLM (with a lag of around 26 months), indicating a delayed adoption process.

The study tested whether multimodality spreads through routine adaptations of text-only checkpoints by analyzing lineage-conditioned transition rates. Using recorded parent-child relations, such as fine-tuning, merging, and quantization, the researchers measured how often text-generation parents produced VLM descendants. showed weak cross-type transfer: only 0.218% of fine-tuning edges from text-generation parents yielded VLM children, with similar low rates for merges (0.104%) and quantization (0.133%). In contrast, the transition matrix revealed that fine-tuning is predominantly task-preserving, with most edges maintaining the same modality, such as text-generation to text-generation. This indicates that direct conversions from text-only to multimodal models are exceptionally rare, challenging the idea of gradual diffusion.

Time-resolved analyses further clarified these dynamics. Monthly estimates of the probability that a text-generation parent would produce a VLM child via fine-tuning remained near zero throughout the observation window, with only transient increases, such as a spike to 0.943% in November 2024. In contrast, when conditioning on VLM parents, the probability of producing VLM children was much higher, often exceeding 75%, showing strong path dependence within multimodal lineages. This asymmetry suggests that the late surge in VLM prevalence within open LLM families is not driven by increasing conversion rates but by rapid expansion within existing VLM lineages after rare founder events.

The research identified a founder-driven mechanism where multimodality enters families through rare integration events. Most VLM releases appear as new roots without recorded parents (approximately 60%), while the remainder are predominantly derived from existing VLM checkpoints. Among fine-tuning edges that produce VLM children, 94.5% originate from VLM parents, compared to only 4.7% from text-generation parents. Founder concentration analyses revealed that a small number of parent models, such as 'naver-clova-ix/donut-base' accounting for 28.2% of VLM-to-VLM edges, dominate downstream derivatives, consistent with classic founder effects where new lineages expand rapidly before diversifying.

These have important for understanding innovation diffusion in AI ecosystems. They explain why multimodal capabilities can appear bursty even as underlying s progress continuously, due to path dependence where early successful VLM founders become key conduits for downstream derivatives. Practically, this means improvements in text-only families may not quickly propagate to multimodal variants without explicit integration work, potentially slowing cross-lineage diffusion. The study also suggests that if more standardized, low-friction ways to attach vision modules are developed, transition rates could rise, but current dynamics are shaped by the high complexity of multimodal integration.

Limitations of the study include reliance on self-reported lineage metadata, which may be incomplete and inflate the share of new roots, and noisy task tags that serve as indicators of intended use rather than ground-truth capability. Time-resolved analyses are constrained by timestamp backfilling starting in March 2022, and family identification uses name-based proxies that might misclassify edge cases. Additionally, the analysis focuses on diffusion patterns in metadata and lineages, not on capability scaling measured by benchmarks, which would require further integration for causal attribution. Despite these bounds, provide a clear picture of multimodal evolution as bottlenecked by rare events and dominated by within-lineage expansion.