Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Misleading error message with SPLASH_URL missing scheme #120

Open
redapple opened this issue Jun 1, 2017 · 0 comments
Open

Misleading error message with SPLASH_URL missing scheme #120

redapple opened this issue Jun 1, 2017 · 0 comments

Comments

@redapple
Copy link
Contributor

redapple commented Jun 1, 2017

Originally appeared on StackOverflow.

When SPLASH_URL setting is missing the http:https:// scheme, scrapy's error is not very helpful

SPLASH_URL = 'localhost:8050'
(...)
2017-06-01 14:44:35 [scrapy.core.scraper] ERROR: Error downloading <GET http:https://example.com/ via http:https://example.com/>
Traceback (most recent call last):
  File "/home/paul/.virtualenvs/scrapy-splash/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1301, in _inlineCallbacks
    result = g.send(result)
  File "/home/paul/.virtualenvs/scrapy-splash/local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 37, in process_request
    response = yield method(request=request, spider=spider)
  File "/home/paul/.virtualenvs/scrapy-splash/local/lib/python2.7/site-packages/scrapy_splash/middleware.py", line 358, in process_request
    priority=request.priority + self.rescheduling_priority_adjust
  File "/home/paul/.virtualenvs/scrapy-splash/local/lib/python2.7/site-packages/scrapy/http/request/__init__.py", line 94, in replace
    return cls(*args, **kwargs)
  File "/home/paul/.virtualenvs/scrapy-splash/local/lib/python2.7/site-packages/scrapy_splash/request.py", line 76, in __init__
    **kwargs)
  File "/home/paul/.virtualenvs/scrapy-splash/local/lib/python2.7/site-packages/scrapy/http/request/__init__.py", line 25, in __init__
    self._set_url(url)
  File "/home/paul/.virtualenvs/scrapy-splash/local/lib/python2.7/site-packages/scrapy/http/request/__init__.py", line 58, in _set_url
    raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url: render.html

Maybe the plugin can detect that the splash URL is wrong and fail with a more actionable message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants