Skip to content

Downloads USPTO patents and finds molecules related to keyword queries

License

Notifications You must be signed in to change notification settings

learningmatter-mit/PatentChem

Repository files navigation

PatentChem

License: MIT DOI

This code downloads the weekly bulk files from the USPTO, selects the patents relevant to chemistry, and queries the chemistry patent claims sections for given keywords to find molecule SMILES related to those keywords.

Installation

conda env create -f environment.yml

Alternatively, the following may be faster:

conda install -c conda-forge mamba
mamba env create -f environment.yml

Example Usage

The scripts should be run in the following order. Steps 1 and 2 only need to be run once, then steps 3 and 4 can be run as many times as desired using different query terms.

Alternatively, we have made a file available on Zenodo with the results of steps 1-2. This file can be downloaded and used as input to steps 3-4 to avoid having to run steps 1-2. We've created a script to download and unzip this file: download_from_zenodo.sh. This download (~75GB) is slow even if you have a fast connection (may take 1-2+ hours due to Zenodo limitations). However, this may still be faster than downloading everything from USPTO. We also recommend this option if you don't have enough hard disk space to store the full dataset in step 1 (see Download File Sizes).

1. Download

Download all USPTO patents for the given years:

python download.py --years 2021 --data_dir /data/patents_data/

The download speed seems to be restricted to ~5-10MB/s by USPTO, which means downloading the full set could require > 4 days if done in series. Alternatively, you can run multiple downloads in parallel. Either way, we recommend starting the downloads in a tmux session to run in the background. For example, you might consider starting several tmux sessions to do the download in parallel using a script similar to download_tmux.sh. Please see Download File Sizes for the approximate size of the files to be downloaded and use caution to avoid filling your whole hard drive.

Additional options:

  • --force_redownload: download files even if they already exist
  • --no_uncompress: do not uncompress the *.zip and *.tar files after selecting the chemistry-related ones (to save space if downloading all years at once)
  • --remove_compressed: remove the original .tar and .zip files for the weekly releases after decompressing them

2. Select

Select the patents related to chemistry based on the presence of CDX files:

python select_chem.py --years 2021 --data_dir /data/patents_data/

Additional options:

  • --remove_compressed: remove all original *.zip and *.tar files after unzipping those relevant to chemistry

3. Search

Provide a list of query terms to search for in the claims sections of the chemistry-related patents:

python search.py --years 2021 --data_dir /data/patents_data/ --naming opd --subject_list "organic electronic" "photodiode" "organic photovoltaic"

4. Clean

Output a yaml file with the SMILES strings found related to the query terms and all of the patents related to each SMILES. Also output a text file with all unique SMILES strings from the query.

python clean.py --years 2021 --naming opd

Additional options:

  • --output_dir: parent directory of --naming if different from .
  • --charged_only: include only charged molecules in output
  • --neutral_only: include only neutral molecules in output
  • --mw_min: include only molecules with molecular weight greater than this
  • --mw_max: include only molecules with molecular weight less than this

Other Useful Commands

  • If you ran download.py with --no_uncompress and now want to uncompress the files: python utils/unzip.py --years 2021 --data_dir /data/patents_data/
  • If you've run select_chem.py but want to reset the data directory to its state after downloading (with unzipped files): python utils/reset.py --years 2021 --data_dir /data/patents_data/
    • This assumes you didn't use --remove_compressed with download.py. If you used --remove_compressed, you'll have to redo the download.py step.
    • Add --remove_uncompressed_only if you want to reset to a state after downloading with --no_uncompress (all files other than the original tar/zip files will be deleted)

Download File Sizes

The following file sizes are taken from the USPTO Bulk Data Storage System using the URLs https://bulkdata.uspto.gov/data/patent/grant/redbook/<YEAR>/ and converting from bytes to GB. The 2023 file size is as of 23 May 2023. Note that these file sizes are for the compressed (zip or tar) files, so the total space required to store this data is larger than what is reported below. Use caution to avoid filling your hard drive to capacity.

Year File Size Units
2001 37.07 GB
2002 37.23 GB
2003 39.20 GB
2004 38.95 GB
2005 30.67 GB
2006 39.90 GB
2007 39.98 GB
2008 42.08 GB
2009 43.82 GB
2010 61.65 GB
2011 68.24 GB
2012 85.42 GB
2013 96.55 GB
2014 111.26 GB
2015 119.98 GB
2016 117.42 GB
2017 124.10 GB
2018 148.02 GB
2019 182.28 GB
2020 192.45 GB
2021 184.41 GB
2022 184.1 GB
2023 80.3+ GB
Total 2.1 TB

Notes

  • For simplicity, clean.py replaces "*" and "Y" substituents from Markush structures with ethyl and O groups. This might not be appropriate for your applications.

Citation

If you use this code, please cite the following manuscript (preprint also available):

@article{subramanian_automated_2023,
	title = {Automated patent extraction powers generative modeling in focused chemical spaces},
	volume = {2},
	issn = {2635-098X},
	url = {https://pubs.rsc.org/en/content/articlelanding/2023/dd/d3dd00041a},
	doi = {10.1039/D3DD00041A},
	number = {4},
	journal = {Digital Discovery},
	author = {Subramanian, Akshay and Greenman, Kevin P. and Gervaix, Alexis and Yang, Tzuhsiung and Gómez-Bombarelli, Rafael},
	year = {2023},
	pages = {1006--1015},
}