Script for data reading/wrangling #25
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- (External) Move all data files to data.world repository (https://data.world/data4democracy/drug-spending) - Remove data/ directory - Correct notebook code to work with data.world as a source
…t downloads Part D data based on notebook
|
Added @mattgawarecki as a reviewer because Python, but this sounds fantastic! Thanks, @dhuppenkothen! |
Changes for incorporation into D4D PR#25
…orts cvs and feather at the moment
|
This PR was getting to be quite large -- I see some small places where we could improve, but I'm not going to let "perfect" get in the way of "great" here. Merged with great pleasure! 👍 |
mattgawarecki
pushed a commit
that referenced
this pull request
Feb 15, 2017
* Data Wrangling for Drug Use Visualizations (#24) * Pulling drug use classes out of the CMS PUF files for categorizing the Plan D data * Fixed directory creating, refactored df, shortened loop * Script for data reading/wrangling (#25) * Fix gitignore to ignore XLSX/ZIP and move CMS data to its own dir * Add drug names back into annual spending data, for ease of use * Forgot to add notebook in last commit * Fix exploration notebook after earlier changes * Add .DS_Store files to .gitignore * Remove Medicare drug spending dataset (migrated to data.world) * Remove data in favor of using data.world - (External) Move all data files to data.world repository (https://data.world/data4democracy/drug-spending) - Remove data/ directory - Correct notebook code to work with data.world as a source * Wrote a helper function that gets data from a URL, wrote function that downloads Part D data based on notebook * Added docstring to function * Added more functions that load data wrangle it * Added argument parser and squashed some bugs * Removed dependence on openpyxl, since Pandas does the trick * Notebook runs * Minor change to command line arg and addition to help string * Move comments into Markdown cells and add CSV output * Added functionality to decide between input/output data formats; supports cvs and feather at the moment * Small bug fix to remove hard-coded directory paths (#28) * Markdown version of goals statement - first draft. * add USP drug classification tidying and data dictionary * Added anchor for citations and superscripted refs * Cleaning drug manufacturer data source: https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Part-B-Drugs/McrPartBDrugAvgSalesPrice/2016ASPFiles.html * move USP Drug Classification to * clarify usp drug classification data dict info * small changes * update links * small changes * download from data.world, clean up functions, add comments * added direct link to the datasets of interest * Restructuring of repo (#36) * Reorganization FTW * Reorganization FTW, part 2 * Add .gitignore * Add READMEs to each subdirectory. Rename data dictionary template (now TEMPLATE) and remove suffix from manufacturer_datadict.md. * Add link to data.world Python client * Update main README to reflect new file structure * Fix link to datadictionaries * Really fix it this time * Fix the other datadictionaries links to overview and template * More streamlining and edits to README * Add @skirmer to maintainers! * Correct directory name for datadictionaries * Add link to objectives doc * Update datadictionaries/README.md to reflect updated repo structure * update files to reflect repo structure changes
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I took the code from the notebook that got merged today, along with this notebook and made a script that can be called from the command line. Of course, the functions can all also be important and called from within python.
It allows the user to decide which data to download (including all), and also includes a helper function to avoid duplicating a lot of code. Examples:
Get help:
python read_data.py -hDownload all data into a specified directory:
python read_data.py -a -d "/path/to/data/directory/"Download Part D data only into the default directory
"../data/":python read_data.py --download-partdDon't download data, but make a file that associates drug names with classes and IDs:
python read_data.py --make-drug-tableI also made a small change to the notebook referenced above, to remove the dependency on
openpyxl, which is unnecessary given that we're importingpandasanyway.Comments/suggestions welcome. :)