Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow spider attr #15

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Allow spider attr #15

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
39740cb
[middleware] allow enabling splash per spider
pawelmhm Apr 3, 2015
9efd145
[middleware] return HtmlResponse to spider
pawelmhm Apr 3, 2015
652fd6e
[scrashtest] add another test spider
pawelmhm Apr 3, 2015
2e7407d
[spider attr] removed html_response
pawelmhm May 26, 2015
bed8998
[tests] refactors tests, adds tests for spider attr
pawelmhm May 26, 2015
a9b5323
Revert "[tests] refactors tests, adds tests for spider attr"
pawelmhm Jun 19, 2015
586ab58
[#15/spider_attribute] adds proper tests
pawelmhm Jun 19, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[spider attr] removed html_response 
* removed process_html (render response in html feature) from this PR
* removed remote request tests with extra spider
  • Loading branch information
@pawelmhm
pawelmhm committed May 26, 2015
commit 2e7407de724c7dce25b1b149bc8c1552cfa86505
31 changes: 0 additions & 31 deletions example/scrashtest/spiders/dmoz_two.py
View file
Open in desktop

This file was deleted.

13 changes: 0 additions & 13 deletions scrapyjs/middleware.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,21 +118,8 @@ def process_response(self, request, response, spider):
self.crawler.stats.inc_value(
'splash/%s/response_count/%s' % (endpoint, response.status)
)
response = self.html_response(response, request)
return response

def html_response(self, response, request):
"""Give user nice HTML response he probably
expects.
"""
data = json.loads(response.body)
html = data.get("html")
if not html:
return response

return HtmlResponse(data["url"], body=html, encoding='utf8',
status=response.status, request=request)

def _set_download_slot(self, request, meta, slot_policy):
if slot_policy == SlotPolicy.PER_DOMAIN:
# Use the same download slot to (sort of) respect download
Expand Down