• Comprehensiveness: |K ∩ W|/|W|
• Relatedness: |Ri |/|Ai |, Ri ∈ Ai, i ∈ {ti , ..., tn }
• Correctness: |C |/|A|, C ∈ A

To train BERT-based model and use it to automatically evaluate the Relatedness and Correctness of the ten KGs, run the command:

nohup python3 /home/huyen/CORD-19-KG/Evaluation/auto_eval_correct_related_BERT_ent.py > /home/huyen/CORD-19-KG/Evaluation/auto_eval_correct_related_BERT_ent.out &
nohup python3 /home/huyen/CORD-19-KG/Evaluation/auto_eval_correct_related_BERT_triple.py > /home/huyen/CORD-19-KG/Evaluation/auto_eval_correct_related_BERT_triple.out &

Our BERT-based models' performance:

precision = tp/(tp+fp)	
recall=tp/(tp+fn)	
F1 =  (2 * Precision * Recall)/Precision + Recall)	
Accuracy = tp/(tp+fn)	

where:

• tp: number of terms* that are shared between the correct answer and the prediction.
• fp: number of terms that are in the prediction but not in the correct answer.
• fn: number of terms that are in the correct answer but not in the prediction. ¹