Skip to content

Price history scraping, archival and plotting for popular online auction market

Notifications You must be signed in to change notification settings

matteopt/bay-scrape

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

bay-scrape

Price history scraping, archival and plotting for popular online auction market.

Demo: GPUs

Interactive plot available at https://robot.emerge.moe/market/gpu.

Demo image

Stages

1. Scraping

Data is scraped via Selenium. What to scrape can be configured via config.py.
The system is not perfect due to ever-changing HTML and a couple interesting issues on the website.
Nonetheless, the output of the previous run can be passed to update the data and keep a full record.

2. Filtering

Following the scraping process, the data is very noisy. There are many unwanted data points.
Filtering uses regular expressions to remove unrelated items and improve accuracy.

See the pre/post-filtering comparison:

Pre/post-filtering comparison

3. Plotting

Plotly is used to plot the data. An HTML file is created.

About

Price history scraping, archival and plotting for popular online auction market

Topics

Resources

Stars

Watchers

Forks

Languages