Some experiments to compare the performances of some pre-trained transformer models on a basic sentiment regression task after fine-tuning them on a sample of the dataset.

A total of 8 transformer models were trained (fine-tuned) on product reviews from the Amazon Review Dataset in the product category Traditional Laptops (Electronics 🡒 Computers & Accessories 🡒 Computers & Tablets 🡒 Laptops 🡒 Traditional Laptops) to predict the star rating of the review given the concatenated summary and text of a review.

A random sample with a size of 10.000 reviews (2.000 for each of the five rating classes 1, 2, 3, 4 and 5 stars) was used to fine-tune each of the pre-trained models on the Laptops reviews data.

The model fine-tuning and evaluation was implemented in Flair. The task was defined as regression task using the TransformerDocumentEmbeddings class (which uses models from huggingface) and the (experimental) TextRegressor class. The maximum number of training epochs for each model was set to 10.

¹: Mean Squared Error
²: Mean Absolute Error
³: Pearson correlation coefficient
⁴: Time to complete training & evaluation (NVIDIA GeForce GTX 1660 Ti)

• pytorch==1.7.1
• cudatoolkit=10.1

• flair
• ujson

The uploaded versions of the training data in this repository are cut off after the first 50 rows of each file, the real training data contains a combined 10.000 rows. The trained model files final-model.pt for each model are omitted in this repository.