Paper link: https://arxiv.org/abs/2306.03168

Authors: Si Wu, David A. Smith

To be appear at the Workshop on Narrative Understanding at ACL 2023!

We will upload a .yml file soon!

Conceptual 12M (CC12M) (Changpinyo et al., 2021): available to download at https://github.com/google-research-datasets/conceptual-12m

Cornell Newsroom Dataset (Grusky et al., 2018): available to download after accepting the data licensing terms https://lil.nlp.cornell.edu/newsroom/download/index.html

Once you get their dataset, run python ./dataset/conceptual_captions/sample_cc12m_captions.py and ./dataset/cornell_newsroom/sample_cornell_sentences.py to generate samples

original.py permuted.py same_img.py backward.py just_noun.py are the scripts to run DALLE on input prompts, it will generate images and output their average CLIP score (aveCLIP). These files are here as a reference, you can reuse the functions in those files to perform a deformance and use DALLE mini to generate images.

Notice that we organize our input and output in this following structure,

caption_file = './dataset/cornell_newsroom/newsroom_samples.txt'
img_output_dir = './img_result/cornell_newsroom/same_img_noun/' 
output_text_dir = './dataset/cornell_newsroom/output_text/same_img_noun/'
poem_stats_path = './stats/cornell_newsroom/same_img_noun/'  + 'clip_stats' + '.csv'

input prompts are in caption_file = './dataset/cornell_newsroom/newsroom_samples.txt' where each line is a prompt,

for each input prompt, the generated image will be in img_output_dir = './img_result/[DATASET_NAME]/[DEFORMANCE_TYPE]/[PROMPT_ID]' ,

then we also save a copy of the deformed text to output_text_dir = './dataset/cornell_newsroom/output_text/same_img_noun/',

finally, we save all the CLIP score information in poem_stats_path = './stats/cornell_newsroom/same_img_noun/' + 'clip_stats' + '.csv'

python analyze_img.py to get image similarity score (imgSim) among the DALLE generated images of the same prompt.

@misc{wu2023composition,
      title={Composition and Deformance: Measuring Imageability with a Text-to-Image Model}, 
      author={Si Wu and David A. Smith},
      year={2023},
      eprint={2306.03168},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Please also feel free to reach out to us for questions and assistance!