A webscraper that scrapes Malaysia eRemit Companies (Wise, Instaremit, MoneyMatch, SunwayMoney) via APIs and stores them in a comma-separated-value (.csv) file.
Some applications for webscraping rates is storing them for future rates predictions and machine learning applications/projects.
This repository is solely for learning purposes on web/api scraping.
- Please respect a website's terms or robots.txt on webscraping.
Ethics in Web Scraping | Towards Data Science
Ensure you have installed the following packages requests and pandas.
pip install requests
pip install pandasClone the Repository
git clone https://github.com/lester-liam/weather-forecast-email.git
In `config.csv`, ensure you added the currencies / rates you want.
Columns Info:
# Remittance Site to Scrape
provider:list = ["Wise", "Instarem", "MoneyMatch", "SunwayMoney"]
country:str # Country Name (non-standard for output)
currency:str # Currency Code as ISO 4217 Format
amount:int # Amount to Convert (integer only)
param1:str # Additional Parameter for Sunway MoneyFor Sunway Money:
Input param1 as Currency Code ISO 4217 format + Country Code as ISO 3166 format.
Example: Singapore Dollar = SGD-SG
The code should run out of the box. However you can also configure the file output locations.
# Stores the Currency/Countries to Query
configFilename:str = "config.csv"
# Final Output Filename
output_filename:str = f"rates_history/rates_history_{datetime.now().strftime("%d%b%y-%H%M")}.csv"main.py: Line 30:
# Read CSV Outputs
wise_rates = pd.read_csv("./output/wise.csv", ...)
instarem = pd.read_csv("./output/instarem.csv", ...)
sunway_money = pd.read_csv("./output/sunwaymoney.csv", ...)
money_match = pd.read_csv("./output/moneymatch.csv", ...)rates_funcs.py: Line 323
# Reset CSV Files:
filenames:list = ["./output/instarem.csv", "./output/moneymatch.csv", "./output/wise.csv", "./output/sunwaymoney.csv"]
for file in filenames:
with open(file, 'w', newline="") as output:
output.write('')rates_funcs.py: Line 332:
# Define processes for each provider
processes = [
Process(..., "./output/instarem.csv")),
Process(..., "./output/sunwaymoney.csv")),
Process(..., "./output/moneymatch.csv"))
]Remittance Websites used in the code.
MoneyMatch
Instarem
Wise
Sunway Money