Codes and datasets for our IJCAI'2022 paper: Targeted Multimodal Sentiment Classification based on Coarse-to-Fine Grained Image-Target Matching
Author
Jianfei Yu & Jieming Wang
We adopt two kinds of datasets to systematically evaluate the effectiveness of ITM.
- Twitter datasets for the TMSC task: the processed pkl files are in floder
./data/Sentiment_Analysis/twitter201x/. The original tweets, images and sentiment annotations can be download from https://drive.google.com/file/d/1PpvvncnQkgDNeBMKVgG2zFYuRhbL873g/view - Image-Target Matching dataset for the two auxiliary tasks: the processed pkl files are in floder
./data/Image_Target_Matching/. The original annotated xml files can be download from Baidu Netdist with code:rm6j. Images of ITM are from twitter2017 dataset.
- pkl files format
{'1': {'iid': '16_05_01_105',
'sentence': 'Safety $T$ has signed a free agent contract with the Cleveland Browns ! # BBN # NFLCats # WeAreUKStill',
'aspect': 'A . J . Stamps',
'sentiment': '2', ## for positive
'relation': '1', ## for related
'boxes': [(182, 9, 853, 856)]
},
...
We use Faster-RCNN to extract region feature as the input feature of images.For the details, you can refer to the original Github. Our processed image feature can be download from Baidu Netdist with code fv25 or GoogleDrive.
python ./tools/extract_feat.py --gpu 0 \
--cfg experiments/cfgs/faster_rcnn_end2end_resnet.yml \
--def models/vg/ResNet-101/faster_rcnn_end2end_final/test.prototxt \
--net data/faster_rcnn_models/resnet101_faster_rcnn_final.caffemodel \
--img_dir ../ITM/data/twitter_images/twitter2017_ori/ \
--out_dir ../ITM/data/twitter_images/twitter2017/ \
--num_bbox 100,100 \
--feat_name pool5_flat
Note that you should change data path.
- Training for ITM
python train.py
--dataset ${i} \
--data_dir ./data/Sentiment_Analysis/ \
--VG_data_dir ./data/Image-Target Matching/ \
--imagefeat_dir ./data/twitter_images/ \
--VG_imagefeat_dir ./data/twitter_images/ \
--output_dir ./log/
- Inference for ITM
python test.py
--dataset ${i} \
--data_dir ./data/Sentiment_Analysis/ \
--VG_data_dir ./data/Image-Target Matching/ \
--imagefeat_dir ./data/twitter_images/ \
--VG_imagefeat_dir ./data/twitter_images/ \
--output_dir ./log/ \
--model_file pytorch_model.bin
- Using the Image-Target Matching dataset means you have read and accepted the copyrights set by Twitter and dataset providers.
- Most of the codes are based on the codes provided by huggingface.