-
Notifications
You must be signed in to change notification settings - Fork 88
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Blacklist domains #94
Comments
I would also log a warning for the first time it happens during a crawl. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I was setuping autoextract in scrapy cloud on a project with crawlera addon. Autoextract queries were routed through crawlera. Idea is to blacklist autoextract domain by default. It may have sense for other services, e.g. spalsh.
It is possible to implement this without adding new options, e.g. adding something to https://github.com/scrapy-plugins/scrapy-crawlera/blob/019987f68345079db176405c9f9fbb155ee26f20/scrapy_crawlera/middleware.py#L32
The text was updated successfully, but these errors were encountered: