KPEval is a toolkit for evaluating your keyphrase systems. 🎯

We provide semantic-based metrics for four evaluation aspects:

• 🤝 Reference Agreement: evaluating the extent keyphrase predictions align with human-annotated references.
• 📚 Faithfulness: evaluating whether each keyphrase prediction is semantically grounded in the input.
• 🌈 Diversity: evaluating whether the predictions include diverse keyphrases with minimal repetitions.
• 🔍 Utility: evalauting the potential of the predictions to enhance document indexing for improved information retrieval performance.

If you have any questions or suggestions, please submit an issue. Thank you!

• [2024/02] 🚀 We have released the KPEval toolkit.
• [2023/05] 🌟 The phrase embedding model is now available at uclanlp/keyphrase-mpnet-v1.

We recommend setting up a conda environment:

conda create -n kpeval python=3.8
conda activate kpeval

Installing the required packages:

• Install torch. Example command if you use CUDA GPUs on linux:

  pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu117
  

• pip install -r requirements.txt

We provide the outputs obtained from 21 keyphrase models in this link. Please run tar -xzvf kpeval_model_outputs.tar.gz to uncompress. Please email [email protected] or open an issue if the link expires.

The execution of all the evaluation aspects are integrated in the run_evaluation.py script. We provide a simple bash script to run the evaluation. You can simply run:

bash run_evaluation.sh [dataset] [model_id] [metric_id]

For example:

bash run_evaluation.sh kp20k 8_catseq semantic_matching

Two log files containing the evaluation results and the per-document scores will be saved to eval_results/[dataset]/[model_id]/. Please see below for the metric_id corresponding to various metrics.

The major metrics supported here are the ones introduced in the KPEval paper.

metric_id is the argument to provide to the evaluation script, and result_field is the field in the result file where the metric's results are stored.

Note: to evaluate utility, you need to prepare the training data using DeepKPG and update the config to point to the corpus.

In addition, we support the following metrics from various previous work:

• New dataset: create a config file at configs/sample_config_[dataset].gin.
• New model: store your model's outputs at model_outputs/[dataset]/[model_id]/[dataset]_hypotheses_linked.json. The file should be in jsonl format containing three fields: source, target, prediction. If you are conducting reference-free evaluation, you may use a placeholder in the target field.
• New metric: just implement it in a new file in the metrics folder. The metric class should inherit KeyphraseMetric. Make sure you update metrics/__init__.py and run_evaluation.py. Also make sure you update the config file in configs with the parameters for your new metrics.

If you find this toolkit useful, please consider citing the following paper.

@article{wu2023kpeval,
      title={KPEval: Towards Fine-grained Semantic-based Evaluation of Keyphrase Extraction and Generation Systems}, 
      author={Di Wu and Da Yin and Kai-Wei Chang},
      year={2023},
      eprint={2303.15422},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}