AI's Hidden Cost Problem: How Simple MLP Models Are Beating Transformers at LLM Efficiency

In the high-stakes world of large language model deployments, a fundamental inefficiency has quietly become a multi-million dollar problem. Contemporary LLM systems typically employ uniform prompting strategies across all query types, applying verbose response patterns to both complex analytical tasks and straightforward factual questions. This one-size-fits-all ology leads to substantial token inefficiency, a concern amplified by the significant cost differential between input and output tokens—the latter commanding 4–8× higher prices across major providers like OpenAI, Google, and Anthropic. According to new research from Bharadwaj Yadavalli, this economic reality creates a pressing need for smarter, more adaptive approaches to LLM prompting that can significantly reduce operational costs without compromising response quality.

Yadavalli's paper, "Dynamic Template Selection: Transformer Approaches for Output Token Generation Optimization," presents a compelling solution to this problem through Dynamic Template Selection (DTS), which adaptively matches response templates to query complexity. The research team compared two routing approaches: a simple MLP that uses pre-computed embeddings and a more complex fine-tuned RoBERTa transformer. Through comprehensive evaluation on 1,000 MMLU questions, they found that the MLP router achieves 90.5% routing accuracy on held-out test data, marginally exceeding RoBERTa's performance (89.5%) despite utilizing 125M fewer parameters. This unexpected result s conventional wisdom about the necessity of complex transformer architectures for classification tasks, suggesting that simpler models can sometimes outperform their more sophisticated counterparts when properly designed and implemented.

Ology behind this breakthrough involves a carefully designed DTS framework consisting of three components: a router that classifies queries into template categories, a template set with five response templates of varying verbosity, and an LLM backend that generates responses using selected templates. The researchers implemented a dual-layer token control mechanism that combines soft prompting (system-level instructions guiding model behavior) with hard token caps (API-level max tokens parameters) to ensure robust token reduction. Templates were designed with specific max tokens values: minimal (50), standard (200), verbose (500), technical (400), and executive (150), with unknown templates defaulting to 1000 tokens for safety. This approach ensures predictable response lengths while maintaining flexibility across different query types and complexity levels.

Empirical from 9,000 production API calls across three major LLM providers reveal the system's impressive performance and cross-provider generalization capabilities. The routing model maintains 90.5% accuracy regardless of the target LLM provider—OpenAI GPT-4, Google Gemini 2.5 Pro, or Anthropic Claude Sonnet—demonstrating that template selection exhibits provider-agnostic properties. While routing accuracy remains consistent, observed token reductions vary from 32.6% to 33.9%, reflecting provider-specific generation characteristics. Gemini shows the highest savings at 33.9%, followed by OpenAI at 33.0%, and Claude at 32.6%. These percentage reductions translate to substantial cost savings when scaled to millions of queries, particularly given that output tokens cost 4–8× more than input tokens across all provider tiers.

Of this research extend far beyond academic interest, offering practical pathways to substantial cost reduction in production LLM deployments. The MLP router introduces minimal latency overhead (approximately 5ms per query) and can be deployed immediately without GPU infrastructure, though it does require external API calls for embedding extraction. In contrast, the RoBERTa architecture offers privacy-preserving offline inference capability but requires GPU infrastructure and longer training times (∼2.5 hours versus ∼1 minute for the MLP). Both architectures demonstrate production-ready performance with sub-percentage-point variations in key metrics, enabling organizations to choose based on their specific operational constraints rather than accuracy differences alone.

Despite these promising , several limitations merit acknowledgment. The MLP router's dependence on OpenAI's text-embedding-3-small model introduces vendor lock-in concerns, and the dataset's focus on academic domains potentially limits insights into system performance on creative writing tasks or specialized professional contexts. Future research directions include investigating online learning capabilities that would allow the router to improve from real usage data, expanding the approach to route between different LLMs entirely (not just templates), and exploring DTS applicability beyond English-language contexts. Additionally, the template design process currently requires manual specification and tuning, suggesting opportunities for automated template optimization based on user feedback and conversation analysis.

From a theoretical perspective, the research provides comprehensive grounding through formal problem formulation with information-theoretic analysis and generalization bounds from statistical learning theory. The paper models DTS as an information transmission problem where queries are compressed through routing to template decisions, with the optimal routing function maximizing mutual information between query embeddings and routing decisions. This theoretical framework helps explain why additional feature engineering doesn't necessarily improve performance for this specific routing task, and why the simpler MLP architecture can achieve comparable or superior to the more complex transformer approach.

The economic impact of these cannot be overstated. For higher-tier models like GPT-4o, Gemini 2.5 Pro, and Claude Sonnet 4, DTS achieves substantial cost savings: approximately $1,646 per million queries for OpenAI, $1,678 for Google, and $2,225 for Anthropic. These savings scale directly with output token pricing multipliers and provider-specific token generation patterns. The research demonstrates that even modest percentage reductions in output token generation can compound into significant economic benefits at production scale, making DTS a compelling solution for organizations seeking to optimize their LLM API costs while maintaining response quality and system reliability.

Looking ahead, the principles underlying this approach extend naturally to any LLM-based system exhibiting multiple response generation strategies with varying computational or economic costs. Practical applications span numerous domains including retrieval-augmented generation systems that could modulate retrieval depth based on query complexity, code generation platforms that adjust test coverage levels according to task criticality, and customer service applications that calibrate response detail to match inquiry sophistication. Each context presents opportunities for intelligent resource allocation through query-aware routing, potentially revolutionizing how organizations deploy and optimize their AI systems across diverse use cases and operational environments.