The overall pipeline as of submission to EMNLP involves the following scripts. Starting with the one that goes through all the downloaded folders and outputs a VIST-style JSON, the rest take in a VIST JSON and perform various operations that also output a VIST-style JSON.
bloom_downloads_to_vist_format.py
takes the raw downloads and makes them into a json with annotations, albums anddedupe_bloom_vist.py
takes in a VIST-style JSON, goes through and collapses albums and stories, Written primarily by Anna Filighera. Takes in a file likebloom_vist_june15.json
and outputsbloom_vist_june15_deduped_june21.json
filter_languages_in_json.py
applies various filters to "quarantine" stories. I've set it up so that you can optionally turn on different filters, I implemented only "expected scripts" and "filter manually". TF-IIF seems doable as well, others are TODO.add_in_storylet_ids_if_they_dont_exist.py
will take in a JSON, search through annotation fields, and add in the storylet_id field. Turns out having some unique value for each annotation is needed for some codebases.bloom_vist_june15_deduped_june21_langfiltered_june22_with_storylets.json
- pipeline_update_json_licenses.py takes in one of the previous JSON files, looks through the albums, and updates the licenses to be noncommercial. Given files like
bloom_vist_june15_deduped_june21_langfiltered_june22_with_storylets.json
it will create a file likebloom_vist_june15_deduped_june21_langfiltered_june22_with_storylets_licenseupdated.json
- split_vist_json.py will take VIST JSON and output splits per language. Given a file like
bloom_vist_june15_deduped_june21_langfiltered_june22.json
It creates files with the same name but appended suffixes_aaa_test.json
and_eng_train.json
, likebloom_vist_june15_deduped_june21_langfiltered_june22_aaa_train.json
precompute_file_uuids_and_hashes.py
: Given a directory of Bloom books, will calculate md5 and uuid for each file and output them. The resultingids_and_hashes.json
is used in the pipeline above.dataset_stats.py
takes the resulting json from the pipeline and outputs various stats, dicts, etc., which I copy-paste into the top ofbloom_captioning.py
(the loading script) for example. For example the_BLOOM_LANGUAGES_ALPHA3_VALID
is generated here.bloom_captioning.py
is the loading script for the captioning set, depends on the dictionaries above, and uses the "quarantine" field.fix_truncated_images.py
is used to solve problems with images that were uploaded to Bloom in a truncated state. Simply put, we open them, save them with gray pixels where the file got truncated, and declare the problem "fixed". Otherwise we get file loading errors downstream.
One problem we ran into was that we upload the files to S3 bucket but there's no easy way to programmatically get the public HTTP URL for the images. The most successful method involves simply setting the bucket to generate a "Manifest" and parsing the keys from that into URLs.
Related scripts:
- copy_images_for_upload_to_s3.py
was the solution to uploading images in the first place. We didn't necessarily want to upload all of our raw downloads due to copyright/rights concerns. So this script takes only some of the images and puts them in a new folder that we can then upload with an s3
commmand line command.
match_aws_keys_with_ids_and_hashes.py
takes the AWS manifest CSV and the ids/hashes JSON and matches things up. It takes in theids_and_hashes.json
above, andaws_manifest.csv
generated by the AWS S3 bucket's inventory feature to calculate the open HTTPs urls for all the images.
list_files_in_public_bucket.py
: just a quick script to see what files are in our bucket.how_many_languages.py:
a quick script to count unique languages in annotation elements in a VIST Jsonget_json_top_level_keys.py
is a useful script for basically just exploring the layout of a JSON by loading it as a Python dict and printing the top-level keys.
The files for this old pipeline are in original_bloom_lm_pipeline
To re-create the data:
- Use boto3 to extract all
*htm
andmeta.json
files to/path/to/books
with a folder per book - Run
python book_parser.py --source /path/to/books --out /path/to/output
, which should create a folder and put in it one .json per book, containing text and metadata from the htm and meta.json above. - To generate clean book,
python book_cleanup.py --source /path/to/extracts/from/2 --out /path/to/out
- To generate language model file for HF
python book_lmset.py --source /path/to/cleanup/files/from/3 --out /path/to/out
Minimal requirements include...
beautifulsoup4
pandas
tqdm
See bloom-parsing-conda-env.yml
for the exact versions used for the EMNLP pipeline.