Multimodal Detection of Hateful Messages Using Visual-Linguistic Pre-Trained Deep Learning Models

PROJECT INTRODUCTION:

Online hateful messages, more commonly known as “hate speech”, have recently become a major social issue. They are absolutely detrimental for both individuals and society. Many online platforms have employed legions of moderators to identify and remove these messages manually, yet such practices are time-consuming, expensive, and often cause mental illness among the reviewers. As a solution, deep learning-based methods are applied automatically to identify and remove hateful messages. However, it is still challenged to tackle another type of “hateful messages” with a format that leverages both text and image to express users’ intents, which were also called “hateful memes”. The difficulty in identifying hateful memes is down to the variable semantic meaning after combining the text and image. In order to effectively detect a hateful meme, the algorithm must possess strong vision and language fusion capability. Our project moves closer to this goal by feeding the visual features of memes generated by the object detection model VinVL into a Transformer-based VL fusion model OSCAR+, followed by a random forest classifier. After fine-tuning, our model achieved a 0.77 AUROC score on the hateful meme detection task (dataset: https://www.kaggle.com/parthplc/facebook-hateful-meme-dataset), an improvement of 0.15 compared to the best baseline method.