Skip to content
/ PWWS Public

Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency

License

Notifications You must be signed in to change notification settings

JHL-HUST/PWWS

Repository files navigation

Probability Weighted Word Saliency(PWWS)

This repository contains Keras implementations of the ACL2019 paper Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency.

Overview

  • data_set/aclImdb/ , data_set/ag_news_csv/anddata_set/yahoo_10 are placeholder directories for the IMDB Review, AG's News and Yahoo! Answer, respectively.
  • word_level_process.pyandchar_level_process.py contain two different prepressing methods of dataset for word-level and char-level, respectively.
  • neural_networks.py contain implementations of four neural networks(word-based CNN, Bi-directional LSTM, char-based CNN, LSTM) used in paper.
  • Use training.pyto train four NN in neural_networks.py.
  • fool.py, evaluate_word_saliency.py, get_NE_list.py,adversarial_tools.pyandparaphrase.pybuild the experiment pipeline.
  • Use evaluate_fool_results.py to evaluate classification accuracy and word replacement rate of adversarial examples generated by PWWS.

Dependencies

  • Python 3.7.1.
  • Versions of all depending libraries are specified in requirements.txt. To reproduce the reported results, please make sure that the specified versions are installed.
  • If you did not download WordNet(a lexical database for the English language), use nltk.download('wordnet') to do so.(Cancel the code comment on line 14 in paraphrase. py)

Usage

  • Download dataset files from google drive , which include
    • IMDB: aclImdb.zip. Decompression and place the folderaclImdb indata_set/.
    • AG's News: ag_news_csv.zip. Decompression and place the folder ag_news_csv indata_set/.
    • Yahoo Answers: yahoo_10.zip. Decompression and place the folder yahoo_10 indata_set/.
  • Download glove.6B.100d.txtfrom google drive and place the file in /.
  • Run training.py or use command likepython3 training.py --model word_cnn --dataset imdb --level word. You can reset the model hyper-parameters in neural_networks.py and config.py.Note that neither this repository nor the paper provides an implementation of char_cnn on IMDB and Yahoo! Answers datasets.
  • Run fool.py or use command likepython3 fool.py --model word_cnn --dataset imdb --level wordto generate adversarial examples using PWWS.
  • Runevaluate_fool_reaults.pyto evaluate adversarial examples.
  • If you want to train or fool different models, reset the argument in training.pyandfool.py.

Result on pretrained model

runs/contains some pretrained NN models, the information of these models are showed as the following table.

We use these pretrained models to generate 1000 adversarial examples with PWWS.

  • test_set means classification accuracy on test set.
  • clean_1000 means classification accuracy on the 1000 clean samples(from test set).
  • adv_1000 means classification accuracy on the adversarial examples corresponding to the 1000 clean samples.
  • sub_rate means word replacement rate defined in Section 4.4.
  • NE_rate means (number of $NE_{adv}$)/(number of substitute word).

If you want to use this model, rename the them or modify the paths to model in the .py files.

data_set neural_network test_set clean_1000 adv_1000 sub_rate NE_rate
IMDB word_cnn 88.792% 86.2% 5.7% 3.933% 21.395%
word_bdlstm 87.472% 86.8% 2.0% 4.206% 11.094%
word_lstm 88.420% 89.8% 10.4% 6.816% 6.548%
AG's News word_cnn 90.526% 89.0% 13.2% 12.308% 30.877%
word_bdlstm 90.711% 89.3% 12.9% 13.494% 27.227%
word_lstm 91.829% 91.4% 18.1% 18.102% 27.374%
char_cnn 88.224% 88.5% 20.0% 11.979% 23.241%
Yahoo! Answers word_cnn 88.427% 96.1% 8.7% 33.067% 12.768%
word_bdlstm 88.876% 94.4% 9.4% 20.752% 7.016%

Contact

  • If you have any questions regarding the code, please create an issue or contact the owner of this repository.

Acknowledgments

About

Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages