LogAn: AI-Powered Log Analysis at Scale

In the high-stakes world of IT support, engineers are often buried under mountains of log data, struggling to pinpoint the root causes of system failures. Traditional s like keyword searches and predefined rules are notoriously inefficient, leading to prolonged downtime and soaring operational costs. Now, a breakthrough from IBM Research offers a lifeline: a log analytics tool that leverages large language models (LLMs) to automate and accelerate issue diagnosis. This innovation, detailed in a recent study, has already been deployed across 70 software products, processing over 2,000 tickets and saving more than 300 man-hours monthly. By running entirely on CPUs without sacrificing accuracy, it addresses the critical of scaling AI in resource-constrained environments, promising to revolutionize how enterprises handle IT incidents.

To overcome the limitations of traditional log analysis, the researchers developed a multi-stage system that begins with preprocessing to reduce data volume. Logs are first consolidated from multiple files into a single, chronologically sorted master file, then subjected to log templatization. This process groups similar log lines into clusters based on templates that abstract variables, such as transforming "PacketResponder 0 for block blk 11 terminating" into a template like "PacketResponder ⟨∗⟩ for block ⟨∗⟩ terminating". From each cluster, a representative log line is randomly selected to form a reduced set, cutting data volume by up to 90%. Next, LLM inferencing is applied only to this representative set using fine-tuned models for three tasks: Golden Signal Classification (GSC), which categorizes logs into signals like error or latency; Fault Category Prediction (FCP), identifying issues in areas like network or memory; and Named Entity Recognition (NER), extracting key entities such as error codes or process IDs. The core innovation, Label Broadcasting, then propagates these inferred labels back to all log lines in their respective clusters, drastically reducing computational demands compared to processing each line individually.

Experimental demonstrate the tool's remarkable efficiency and effectiveness. In tests on real-world datasets from domains like finance and security, Label Broadcasting reduced LLM inference time by approximately 99.7% compared to traditional s, processing 170,000 log lines in just seconds instead of hours. For instance, on a dataset of 425,000 log lines from a financial application, the tool condensed it to a representative set of 74 lines, achieving a 99.9% reduction without compromising insight quality. Output comparison showed that over 98% of predictions matched those from full-scale inference, with minimal quality degradation in rare cases. A case study highlighted its practical utility: when a client reported unexplained application terminations, the tool's reports identified error spikes and causal relationships, such as latency leading to session suspensions, enabling rapid diagnosis that manual s would have missed.

Of this research extend far beyond IBM's internal use, offering a blueprint for cost-effective AI integration in IT operations worldwide. By enabling LLMs to run efficiently on CPUs, the tool democratizes advanced analytics for organizations without GPU resources, potentially reducing manpower costs by an estimated $15,444 per month per deployment. Its ability to generalize across diverse software domains—from data analytics to inventory management—means it can adapt to various industries without retraining, enhancing scalability. Moreover, the insights generated, such as causal graphs showing how errors propagate, empower support engineers to make data-driven decisions faster, reducing cognitive load and improving system reliability. This approach could set a new standard for AIOps (AI for IT operations), encouraging broader adoption in sectors like cybersecurity and cloud computing where log analysis is critical.

Despite its successes, the tool has limitations that highlight areas for future work. User feedback from deployment revealed that 53.24% of respondents did not find it useful, often due to data quality issues, such as logs with embedded JSON objects that disrupt templatization, or domains like hardware management where the models lack training data. Additionally, the tool does not determine root causes or recommend fixes, relying instead on engineers to interpret the highlighted log lines. Performance can also degrade with extremely large dumps, as seen in microservice-based applications generating gigabytes of data, leading to processing delays or crashes. These s underscore the need for improved preprocessing techniques and domain adaptation, but the overall impact—saving 316 man-hours over 15 months—proves its value as a foundational step toward fully autonomous IT support systems.