-
Notifications
You must be signed in to change notification settings - Fork 7
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
4733cf9
commit 7659f77
Showing
1 changed file
with
30 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,31 @@ | ||
# Wikipedia-Article-Scraper | ||
A full Python package that allows users to search for a Wiki article, scrape it and have features for text analytics. | ||
A full Python package that allows users to search for a Wiki article, scrape it and have features for text analytics. <br> | ||
Please feel free to download and use it!!! | ||
|
||
## wikiscrape Package (can download wikiscrape.py and import it) | ||
### This Python code can be used to search for a Wikipedia Article and do text analytics on it | ||
<br> | ||
To save a wikiscrape object, simply type: <br> | ||
import wikiscrape <br> | ||
var = wikiscrape.wiki('Search Article') <br> | ||
e.g. paris = wikiscrape.wiki('PArIs','Yes','french','Yes') means to search for the article Paris, auto format to proper case (Yes for 2nd argument, default Yes), search for French wikipedia (french for 3rd argument, default No) and apply nltk stoplist for french (Yes for 4th argument, default No) | ||
### Full capabilities of wikiscrape package includes: | ||
1. Able to search in multiple languages | ||
2. Give suggestions on search terms if search is ambiguous <br> | ||
3. Gives a short summary (2 paragraphs) of the article if it is retrieved successfully <br> | ||
4. Retrieve full text or exact number of paragraphs in string output for data pipeline <br> | ||
5. var.HELP() for the full list of functions available <br> | ||
6. Basic error handling, including checking data type of arguments and reverting to defaults if errorneous args are given <br> | ||
|
||
### Text Analytics capabilities include: | ||
1. A frequency counter on the most common words in the Wikipedia article (after omitting common English words, or stoplist from NLTK for foreign languages). Can also find the Nth % of most common words, where 0 =< N =< 100. <br> | ||
2. A graph plot of the most common words in the Wikipedia article <br> | ||
3. A graph plot on the most frequent Years mentioned in the article, to understand the Years of interest of the article <br> | ||
4. A summary on the total number of words and total number of unique words after implementing the stoplist of common words. <br> | ||
|
||
#### Libraries used: requests, bs4, collections, matplotlib, re, os, nltk | ||
|
||
#### Updates: | ||
1. 26 May 2019 - Added plotyear() function to plot the most frequent years mentioned | ||
2. 9 June 2019 - Added markdown for explaination and added comments in the code for understanding <br> | ||
3. For any questions or suggestions, please contact me at my Github account - https://github.com/kohjiaxuan/ <br> |