Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scrapy运行无日志,gerapy请求则会报错 #263

Open
Spider-sun opened this issue Apr 8, 2023 · 1 comment
Open

scrapy运行无日志,gerapy请求则会报错 #263

Spider-sun opened this issue Apr 8, 2023 · 1 comment
Assignees
Labels

Comments

@Spider-sun
Copy link

2023-04-08T15:21:06+0000 [twisted.web.server.Request#critical]
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/twisted/web/server.py", line 292, in render
body = resrc.render(self)
File "/usr/local/lib/python3.11/site-packages/twisted/web/_template_util.py", line 194, in render
self.d.addCallback(self._cbChild, request).addErrback(self._ebChild, request)
File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 532, in addCallback
return self.addCallbacks(callback, callbackArgs=args, callbackKeywords=kwargs)
File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 512, in addCallbacks
self._runCallbacks()
--- ---
File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 892, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/usr/local/lib/python3.11/site-packages/twisted/web/_template_util.py", line 200, in _cbChild
request.render(resource.getChildForRequest(child, request))
File "/usr/local/lib/python3.11/site-packages/twisted/web/server.py", line 292, in render
body = resrc.render(self)
File "/usr/local/lib/python3.11/site-packages/twisted/web/_auth/wrapper.py", line 198, in render
return super().render(request)
File "/usr/local/lib/python3.11/site-packages/twisted/web/resource.py", line 268, in render
return m(request)
File "/usr/local/lib/python3.11/site-packages/twisted/web/static.py", line 657, in render_GET
producer.start()
File "/usr/local/lib/python3.11/site-packages/twisted/web/static.py", line 770, in start
self.fileObject.seek(self.offset)
builtins.OSError: [Errno 22] Invalid argument

2023-04-08T15:21:06+0000 [twisted.python.log#info] "172.17.0.1" - - [08/Apr/2023:15:21:05 +0000] "GET /logs/VulSpiders/cnnvdSpider/f7a99998d62011ed94c70242ac11000a.log HTTP/1.1" 500 95 "-" "python-requests/2.28.1"

@Spider-sun
Copy link
Author

这个问题是views.py中的job_log这个函数在发起请求时的请求头参数headers={'Range': 'bytes=-1000'}引起的,当日志信息为空时,发起请求获取最后1000字节的数据,会使scrapyd发生异常,这应该是scrapyd的问题

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants