AI Generates Text Four Times Faster

Large language models like GPT-4 have transformed technology but face a critical bottleneck: they generate text one token at a time, consuming immense energy and limiting speed. Researchers from Tencent and Tsinghua University have introduced Continuous Autoregressive Language Models (CALM), a framework that overcomes this by predicting chunks of text as single vectors, reducing generation steps by a factor of K—for example, cutting steps by 75% when K=4. This shift not only accelerates AI but also addresses environmental concerns tied to high computational demands.

The key finding is that CALM reframes next-token prediction as next-vector prediction. By compressing a group of K tokens into a dense vector using a high-fidelity autoencoder, the model reconstructs original tokens with 99.9% accuracy. This allows the AI to process sequences of vectors instead of individual tokens, fundamentally improving efficiency without sacrificing performance. In experiments, a CALM model with 371 million parameters achieved results comparable to a 281-million-parameter baseline while using 34% fewer inference computations.

Methodologically, the team developed a lightweight autoencoder that maps token chunks to vectors, ensuring robustness through techniques like variational regularization and dropout to prevent representation collapse. They employed an energy-based generative head for single-step vector prediction, avoiding iterative processes that could reintroduce bottlenecks. Since traditional metrics like perplexity are inapplicable without explicit likelihoods, they introduced BrierLM, a novel evaluation metric based on the Brier score, which provides unbiased assessment of model capabilities using only samples.

Results from tests on datasets like WikiText-103 show CALM's superior performance-compute trade-off. For instance, with K=4, it delivered similar BrierLM scores to strong baselines but at significantly lower cost. The framework also includes a likelihood-free temperature sampling algorithm, enabling controlled generation—such as adjusting output diversity—without access to probability distributions, though it requires more samples at lower temperatures for accuracy.

In real-world terms, this innovation could make advanced AI more affordable and sustainable, reducing the carbon footprint of data centers and enabling faster applications in chatbots, content creation, and research. By scaling the bandwidth of predictive steps rather than just model size, CALM opens a new pathway for efficient AI development, potentially democratizing access to cutting-edge language technologies.

Limitations noted in the paper include challenges at very low temperatures, where sampling efficiency drops, and the current autoencoder's focus on reconstruction over semantic structure. Future work could explore context-aware designs and integrated architectures to further enhance robustness and capability.