AI Cuts Response Times by Reusing Parts of Answers

Large language models (LLMs) are increasingly used in interactive applications like code assistants and data generation tools, where speed and cost are critical. However, existing caching s often struggle when requests are similar but not identical, forcing systems to regenerate entire responses even when only small changes are needed. This inefficiency can lead to higher latency and increased computational costs, limiting the scalability of AI services in real-world settings.

Researchers have developed StepCache, a new caching layer that addresses this gap by reusing responses at the step level rather than as whole outputs. The system works by breaking down previous answers into ordered steps, such as individual paragraphs or structured data units, and then verifying each step against new requests using lightweight, task-specific checks. When a step passes verification, it is reused directly; if it fails, only that step and its dependents are regenerated through a process called selective patching. This approach allows StepCache to handle localized changes, like modifying a variable in a math problem or adding a key to a JSON schema, without recomputing the entire solution.

Ology behind StepCache involves several key components. First, the system segments responses into steps using heuristics like paragraph boundaries or explicit enumerations, with task-aware segmentation for structured outputs like JSON. For retrieval, StepCache computes prompt embeddings and uses approximate nearest-neighbor search to find the best-matching cached request. Verification is performed with rule-based checks: for linear equations, it parses constants and variables to ensure consistency, while for JSON, it validates syntax and required keys. When failures occur, StepCache applies contiguous block patching for math tasks or strict structured patching for JSON, with a conservative skip-reuse policy that falls back to full regeneration if reuse is predicted to be unproductive, such as when core semantics change.

From a CPU-only micro-benchmark demonstrate significant improvements. Averaged over three seeds with 222 evaluation requests per seed, StepCache reduced mean latency from 2.13 seconds to 0.67 seconds and median latency from 2.42 seconds to 0.01 seconds. Total token usage dropped by approximately 24%, from 36.1k to 27.3k tokens. Crucially, correctness improved from 72.5% to 100% under both task-specific checks and a stitched-output integrity check. The breakdown shows that 79.7% of requests took the fast reuse-only path, 5.4% required patching, and 14.9% triggered skip-reuse, with specific tasks like JSON key changes forcing patching but maintaining accuracy.

Of StepCache are substantial for real-world AI deployments. By enabling granular reuse, it can lower operational costs and improve user experience in applications where prompts share common structures but vary in details, such as in automated reporting or educational tools. The system's backend-agnostic design means it can be integrated with existing serving engines without modification, offering a drop-in optimization. However, limitations include its reliance on task-aware verifiers, which are currently implemented only for math and JSON tasks; extending to open-ended text requires more costly verification s. Future work will focus on GPU evaluations, integration with production traces, and enhancing verifiers for broader applications.