Several fixes #14
Merged
Several fixes #14
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ot being added. Refactored the code to have more controls on query execution and log generation.
…a0 for Università. Since MySQL DB is in utf8mb4 and python3 support utf8 natively and the compressed xml files from pubmed are in UTF8, there is no need to force an encode.
- The abstract was containing some XML tags. I supposed that was not intended, so I get rid of them and joined just the text of the various abstract for a specific article. Fix MrMimic#8 - The pub_date month field was not read properly. It can be either expressed as a number '02' or as text like 'Feb'. Fix MrMimic#9 - Citation owner and citation status from the medlinecitation tag were not being extracted correctly. Fix MrMimic#10 - Apparently an article can have multiple language specified in the XML. For the sake of cleanliness of the generated DB I made sure to report all the languages. Fix MrMimic#11 - Apparently an article can have multiple citationsubset specified in the XML. For the sake of cleanliness of the generated DB I made sure to report all the values. Fix MrMimic#12 - Mesh descriptor regex is not capturing long IDs. There are some descriptive_ui like D000074606 for 'Smoking prevention'. Fix MrMimic#13 - The DB field type in the medline_comments_corrections table is not long enough. Increased from 20 to 25. Ran against a couple of xml files and seems to be solid. It may need further testing though. For the moment Fix MrMimic#7 - Making sure to employ utf8 when extracting the xml file from the gz archive
Always build absolute paths
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi!
Thanks for the great work you carry on doing. I am using this tool to have an up-to-date DB on which run a set of analysis.
While using it, I discovered a couple of unexpected behaviours that I reported. Most of them are issues that I fixed in my fork.
Two or them are more considerations than real issues. Feel free to discuss or ignore.
I am making this pull request in case you agree on the fixes I applied in my fork. Otherwise, I will using my branch until the tool got another fix for these issues.
Again, many thanks for your work. It is much appreciated.