KPEval is a toolkit for evaluating your keyphrase systems. 🎯

We provide semantic-based metrics for four evaluation dimensions:

• 🤝 Reference Agreement: evaluating the extent keyphrase predictions align with human-annotated references.
• 📚 Faithfulness: evaluating whether each keyphrase prediction is semantically grounded in the input.
• 🌈 Diversity: evaluating whether the predictions include diverse keyphrases with minimal repetitions.
• 🔍 Utility: evalauting the potential of the predictions to enhance document indexing for improved information retrieval performance.

If you have any questions or suggestions, please submit an issue. Thank you!

• [2024/02] 🚀 We have released the KPEval toolkit.
• [2023/05] 🌟 The phrase embedding model is now available at uclanlp/keyphrase-mpnet-v1.

We recommend setting up a conda environment:

conda create -n kpeval python=3.8
conda activate kpeval

Installing the required packages:

• Install torch. Example command if you use CUDA GPUs on linux:

  pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu117
  

• pip install -r requirements.txt

We provide the outputs obtained from 21 keyphrase models in this link. Please run tar -xzvf kpeval_model_outputs.tar.gz to uncompress. Please email [email protected] or open an issue if the link expires.

The execution of all the evaluation aspects are integrated in the run_evaluation.py script. We provide a simple bash script to run the evaluation. You can simply run:

bash run_evaluation.sh [dataset] [model_id] [metric_id]

For example:

bash run_evaluation.sh kp20k 8_catseq semantic_matching

Two log files containing the evaluation results and the per-document scores will be saved to eval_results/[dataset]/[model_id]/. Please see below for the metric_id corresponding to various metrics.

The major metrics supported here are the ones introduced in KPEval.

metric_id is the argument to provide to the evaluation script, and result_field is the field in the result file where the metric's results are stored.

Note: to evaluate utility, you need to prepare the training data using DeepKPG and update the config to point to the corpus.

In addition, we also support the following metrics:

• New dataset: create a config file at configs/sample_config_[dataset].gin.
• New model: store your model's outputs at model_outputs/[dataset]/[model_id]/[dataset]_hypotheses_linked.json. The file should be in jsonl format containing three fields: source, target, prediction. If you are conducting reference-free evaluation, you may use a placeholder in the target field.
• New metric: just create a new file in the metrics folder. The metric class should inherit KeyphraseMetric. Make sure you update metrics/__init__.py and run_evaluation.py. Also make sure you update the config file in configs with the parameters with your new metrics.

If you find this toolkit useful, please consider citing the following paper.

@article{wu2023kpeval,
      title={KPEval: Towards Fine-grained Semantic-based Evaluation of Keyphrase Extraction and Generation Systems}, 
      author={Di Wu and Da Yin and Kai-Wei Chang},
      year={2023},
      eprint={2303.15422},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}