Repository for the paper: Log Parsing with Prompt-based Few-shot Learning

Abstract: Logs generated by large-scale software systems provide crucial information for engineers to understand the system status and diagnose problems of the systems. Log parsing, which converts raw log messages into structured data, is the first step to enabling automated log analytics. Existing log parsers extract the common part as log templates using statistical features. However, these log parsers often fail to identify the correct templates and parameters because: 1) they often overlook the semantic meaning of log messages, and 2) they require domain-specific knowledge for different log datasets. To address the limitations of existing methods, in this paper, we propose LogPPT to capture the patterns of templates using prompt-based few-shot learning. LogPPT utilises a novel prompt tuning method to recognise keywords and parameters based on a few labelled log data. In addition, an adaptive random sampling algorithm is designed to select a small yet diverse training set. We have conducted extensive experiments on 16 public log datasets. The experimental results show that LogPPT is effective and efficient for log parsing.

An overview of LogPPT

LogPPT consists of the following components:

1. Adaptive Random Sampling algorithm: A few-shot data sampling algorithm, which is used to select K labelled logs for training (K is small).
2. Few-shot Data Sampling: An adaptive random sampling based method for selecting K labelled logs for training (K is small).
3. Prompt-based Parsing: A module to tune a pre-trained language model using prompt tuning for log parsing

1. Python 3.8
2. torch
3. transformers
4. ...

To install all library:

$ pip install -r requirements.txt

To download the pre-trained language model:

$ cd pretrained_models/roberta-base
$ bash download.sh

$ python fewshot-sampling.py

dataset=Apache
shot=32
trf="datasets/${dataset}/${shot}shot/1.json"
tef="datasets/${dataset}/test.json"
python train.py --mode prompt-tuning --train_file ${trf} \
    --validation_file ${tef} \
    --model_name_or_path "./pretrained_models/roberta-base" \
    --per_device_train_batch_size 8 \
    --learning_rate 5e-5 \
    --lr_scheduler_type polynomial \
    --task_name log-parsing \
    --num_warmup_steps 20 \
    --max_train_steps 200 \
    --log_file datasets/${dataset}/${dataset}_2k.log_structured.csv \
    --shot $shot \
    --dataset_name ${dataset} \
    --task_output_dir "outputs"

The parsed logs (parsing results) are saved in the outputs folder.

For the descriptions of all parameters, please use:

python train.py --help

python benchmark.py

** Implementations for baselines are adopted from Tools and Benchmarks for Automated Log Parsing, and Guidelines for Assessing the Accuracy of Log Message Template Identification Techniques.

• Accuracy:

• Robustness:

Robustness across different log data types

Robustness across different numbers of training data

• Accuracy on Unseen Logs:

Accuracy on Unseen Logs

Running time of different log parsers under different volume

• We exclude the Virtual Label Token Generation module and let the pre-trained model automatically assign the embedding for the virtual label token “I-PAR”. To measure the contribution of the Adaptive Random Sampling module, we remove it from our model and randomly sample the log messages for labelling.

Ablation Study Results

• We vary the number of label words from 1 to 16 used in the Virtual Label Token Generation module.

Results with different numbers of label words

We compare LogPPT with fine-tuning, hard-prompt, and soft-prompt.

• Effectiveness:

Accuracy across different tuning methods

• Efficiency:

Parsing time across different tuning methods

• PTA: The ratio of correctly identified templates over the total number of identified templates.

• RTA: The ratio of correctly identified templates over the total number of oracle templates.

Full results with 32shot