Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

allow SPLASH_SLOT_POLICY setting to take a path #163

Open
iAnanich opened this issue Jan 19, 2018 · 1 comment
Open

allow SPLASH_SLOT_POLICY setting to take a path #163

iAnanich opened this issue Jan 19, 2018 · 1 comment

Comments

@iAnanich
Copy link
Contributor

As mentioned under #157 in this comment: this feature must be backward compatible.

@iAnanich
Copy link
Contributor Author

iAnanich commented Jan 19, 2018

I have an idea to create slot_policy.py module with variables equivalent to middleware.SlotPolicy attributes and refactor SlotPolicy's attributes to reference slot_policy.py's variables.

To get slot policy by path middleware.SplashMiddleware._set_download_slot method can be refactored to first search slot_policy argument value in known policies and then attempt to import object using scrapy.utils.misc.load_object function.

Is it better to use if-elif-else for known policies and then make an attempt to import, or if given slot_policy is unknown make attempt to import and then if-elif-else over known slot policies?

Or maybe it's better to extend load_object function to resolve attributes to, for example by adding : character before the name of the attribute. In our case 'per_domain' slot policy setting can be set as 'scrapy_splash.middleware.SlopPolicy:PER_DOMAIN' string.

New tests in test_middleware.py must be created to test different SPLASH_SLOT_POLICY values.

The new feature must be mentioned in README.rst.

  • refactor _set_download_slot
  • test different setting's values
  • update README.rst

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants