Hue: A User-Adaptive Parser for Hybrid Logs

Dec 5, 2023

Speakers

About

Log parsing, which extracts log templates from semi-structured logs and produces structured logs, is the first and the most critical step in automated log analysis. While existing log parsers have achieved high accuracy on publicly available log datasets, they suffer from two major limitations by design. First, all existing parsers do not natively support hybrid logs that consist of both single-line logs and multi-line logs (\textit{e.g.,} Java Exception and Hadoop Counters). Second, most existing parsers fall short in integrating expert domain knowledge during parsing, which makes them hard to identify ambiguous log templates in complex real-world environments. To address these issues, this paper proposes \textit{Hue}, the first log parsing approach for hybrid logs. Hue can parse both hybrid and single-line logs in an online manner, which effectively leverages both patterns in the incoming log messages and domain knowledge from the experts. Specifically, Hue converts each log message to a sequence of special wildcards using a key casting table and then conducts line aggregation and pattern extraction. In addition, Hue can effectively utilize user feedback via a novel merge-reject strategy, which makes it possible to quickly adapt to complex and changing log templates. We evaluated Hue on three hybrid log datasets and sixteen widely-used single-line log datasets (\textit{i.e.,} Loghub). The results show that Hue achieves an average accuracy of 0.845 on hybrid logs, which largely outperforms the best results (0.563) obtained by existing parsers. Hue also exhibits SOTA accuracy and efficiency on single-line log datasets. Furthermore, Hue has been successfully deployed in a real production environment for daily hybrid log parsing.

Organizer

Store presentation

Should this presentation be stored for 1000 years?

How do we store presentations

Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%

Sharing

Recommended Videos

Presentations on similar topic, category or speaker

Interested in talks like this? Follow ESEC-FSE