Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
Added links
  • Loading branch information
pigivinci committed May 30, 2022
1 parent 9406313 commit 18f4e07
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ Don't insert rules for cleaning prices or numeric fields: formats change over di
Load the fewer pages you can. Try to see if the fields you need are all available from product catalogue pages and try avoiding enter the single product page.
#### 2.5. Ip rotation
One of the most basic actions that a target website can take against web scraping is to ban IPs that make too many requests in a certain timeframe. Given that the web scraping activity must not interfere with the website functionality and operations, if this is happening to your scrapers, you might consider splitting its execution from several machines or route it via proxies.
Nowadays there are plenty of proxy vendors on the market and also proxies for every need, we'll go in-depth in this section.
Nowadays there are plenty of proxy vendors on the market and also proxies for every need, we'll go in-depth [in this section](https://github.com/reanalytics-databoutique/webscraping-open-project/blob/main/Pages/Services/Proxies.md).

### 3. Tools
#### 3.1. Headless python scrapers
Expand Down

0 comments on commit 18f4e07

Please sign in to comment.