Automatically Scrape a HUGE amount of url at scale

[benjaminelkrieff]

Hello Everyone.

I am currently working on a project whose goal is to create scrapers in a dynamic way and then being able to process a huge amount of urls at scale.

For example, I have in DDS two websites: www.xxx.it associated with Scraper IT and www.xxx.ca associated with Scraper CA.

I want to send an infinite amount of urls to DDS, and I want all the urls in the form xxx.it/* to be scraped by Scraper IT, and all the urls in the form xxx.ca/* to be scraped by Scraper CA. I want this to be automatically inferred and done by DDS.

Is there a way to achieve this with the current implementation ?