-
Notifications
You must be signed in to change notification settings - Fork 451
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add option to return Htmlresponse generated from content of 'html' key in splash response #14
Comments
Hey @pawelmhm - there is now some magic in scrapy-splash master branch which sets response.body, response.url, response.headers, handles cookies and allows to use response.xpath on JSON splash responses; check the updated README. Feedback is welcome! |
This should be fixed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When processing splash response we could just replace response from splash with new HTMLResponse generated from html in splash response. This way user will not have to worry about generating Html response in spider callback, she can just forget about js rendering - when splash will be enabled she will just get normal target response with JS rendered (with proper url, html etc). User can just use normal response.xpath on this response without doing anything extra. I think rendering html would probably be most common use case for splash middleware so it's worth enabling this by default.
If there will be some other keys in splash response (other then html) we could pass them in meta perhaps? .har content could be converted to response headers
The text was updated successfully, but these errors were encountered: