Explanation:

• Train the model on some dataset (e.g. Fake News Kaggle Competition, or Covid Fake News Dataset)
• Let's embed the whole fact-checking dataset into the same space (not the embeddings, but the centroid of the embeddings).
• Let's try to predict the news to be fake or true.
• After we've predicted let's use the embedding centroid and extract k nearest neighbors from the knowledge base.
• Let's see how well this nearest neighbors match the original fact-checking articles (this we will measure by max/average cosine/euclidian similarity of the neighbors embedding with the original fact-checking articles, and by the number of exact matches, e.g. recall and precision).

Notes: we should actually compare this with S-BERT. For now we just use Mean of BERT embeddings as a centroid. We should also think how we can deal with the fact that to use concept space similarity we need to use N knowledge bases - one for each label. Since otherwise vectors that fall in different spaces will be compared (since we force them to be orthogonal only with those from the different concept spaces).

tp / (tp + fp) = precision (how well we identify true explanation) tp / (tp + fn) = recall (how well we distinguish true explanation from false explanation)

Implement the Eraser benchmark and test it for the eraser datasets. Test the benchmark for the Hate-Xplain dataset.

• Train the model with the dataset, since the documents are too long, we would be sampling only a part of the document with annotated rationale. ??? How to sample the rationale? (check this with the original paper) What if we sample some part of the document that is not relevant and also the part that is relevant and concatenate them?
• Take the query and use it as the text_a, and add sampled text as text_b.
• Remove normalization from space model concept space.

• Precision
• Recall
• F1 Score (IOU)
• Accuracy
• Jaccard Index

Calculate those based on the rationale and the predicted rationale (this is the token predictions for the space model).

Use binary cross-entropy with logits here as the metric.

• Exponentially distribute the weights of the true rationale to the adjacent tokens.
• Softmax the updated weights vector.

Recall:

• Multiply hard predicted rationale with true rationale.
• Sum the resulting rationale vectors.
• Create a histogram of the resulting sums (the more in the higher region - the better).

Precision:

• Exponentially distribute the weights of the predicted rationale to the adjacent tokens.
• Multiply predicted rationale by the true hard rationale.
• Sum the resulting rationale vectors.
• Create a histogram of the resulting sums (the more in the higher region - the better).

...

• test on the additional dataset zero-shot when trained with rationale vs without rationale.

TODO:

• !!! Train only last classifier layer for the space model.
• https://hatebase.org/search_results
• Rational loss to space model