Update 23 June 2024: All metrics in Development mode has been evaluated under closest mode which makes a decision of the result class by relying on the first entry of the label.

Update 11 June 2024: Added evaluation mode that counts first label entry. See eval-mode parameter key.

This repository assess the LLMs reasoning capabilities in Targeted Sentiment Analysis on RuSentNE dataset proposed as a part of the self-titled competition.

In particular, we use pre-treained LLMs for the following datset splits:

1. 🔓 Development~~~~
2. 🔒 Final

The following reasoning we use [quick-cot] to experiment with:

• Instruction Prompts
• Chain-of-Thoughts (THoR)

All the sqlite results are stored in contents table.

Option 1. You may use sqlitebrowser for accessing the results for exporting into CSV.

Option 2. Use sqlite2csv.py script implemented in this repository.

This is an open-access dataset split (sentiment labels available) utilized for the development stage and could be used anyone in evaluation checks.

Dataset: valiation_data_labeled.csv

* -- denotes evaluation in first-entry mode (seeking for the first entry).

This leaderboard and obtained LLM answers is a part of the experiments in paper: Large Language Models in Targeted Sentiment Analysis in Russian.

Dataset: final_data.csv

If you find the results and findings in Final Results section valuable 💎, feel free to cite the related work as follows:

@misc{rusnachenko2024large,
      title={Large Language Models in Targeted Sentiment Analysis}, 
      author={Nicolay Rusnachenko and Anton Golubev and Natalia Loukachevitch},
      year={2024},
      eprint={2404.12342},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}