Li, Teng; Zhang, Shengkai; Feng, Yebo; Xu, Jiahua; Ma, Zhuo; Shen, Yulong; Ma, Jianfeng; (2025) Heuristic-based Parsing System for Big Data Log. In: Proceedings - GLOBECOM 2024 - 2024 IEEE Global Communications Conference. (pp. pp. 2329-2334). IEEE: Cape Town, South Africa.

Preview

Text globecom_Heuristic_based_Parsing_System_for_Big_Data_Log.pdf - Accepted Version Download (548kB) | Preview

Abstract

Logs play a crucial role in recording valuable system runtime information, extensively utilized by service providers and users for effective service management. A typical approach in service management, based on log analysis, involves parsing the original log messages initially presented in an unstructured format. Subsequently, a data mining model is employed to extract critical system behavior information, aiding in service management. As the volume of logs rapidly increases, training models using current log resolution methods post-log collection becomes excessively time-consuming, leading to decreased accuracy. Manual analysis of extensive logs is both time-intensive and inefficient. This article introduces Aclog, an automated log parsing tool tailored for large-scale log analysis, storage, and management. Aclog operates by storing and managing logs in a structured and unified format, thereby offering a cohesive database for comprehensive log auditing of computing systems. Key components of Aclog encompass the log updater, log parser, log storage, and log querier. In this paper, we utilize a realworld, large-scale public log dataset to showcase the capabilities of Aclog. We evaluate the log files generated by ten popular systems.

Download activity - last month

Export as XMLCSVJSON

Download activity - last 12 months

Export as XMLCSVJSON

Downloads by country - last 12 months

Export as JSONXMLCSV

Archive Staff Only

View Item