Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature - Stop long running scrapers. #158

Open
wants to merge 8 commits into
base: master
Choose a base branch
from

Conversation

knmn2000
Copy link

@knmn2000 knmn2000 commented Aug 2, 2020

Introduced a button on the jobs page of scrapydweb, which sends a stop signal to all the currently running scrapers that have been running longer than a certain amount of time, which can be specified in the settings.

This will be especially useful in cases where the project contains hundreds of running spiders.

@pep8speaks
Copy link

Hello @knmn2000! Thanks for opening this PR. We checked the lines you've touched for PEP 8 issues, and found:

Line 87:9: E116 unexpected indentation (comment)

Line 93:13: E722 do not use bare 'except'

Line 68:121: E501 line too long (158 > 120 characters)

Line 127:17: E126 continuation line over-indented for hanging indent
Line 144:17: E126 continuation line over-indented for hanging indent

Line 68:121: E501 line too long (158 > 120 characters)

Line 83:13: W503 line break before binary operator
Line 86:13: W503 line break before binary operator
Line 87:13: W503 line break before binary operator
Line 88:13: W503 line break before binary operator
Line 401:13: W503 line break before binary operator
Line 617:1: W391 blank line at end of file

Line 61:13: E116 unexpected indentation (comment)
Line 139:101: E226 missing whitespace around arithmetic operator
Line 217:53: E226 missing whitespace around arithmetic operator
Line 224:48: E226 missing whitespace around arithmetic operator

Line 52:5: E116 unexpected indentation (comment)
Line 53:5: E116 unexpected indentation (comment)
Line 54:5: E116 unexpected indentation (comment)

Line 63:9: E722 do not use bare 'except'
Line 66:13: E722 do not use bare 'except'
Line 82:13: E722 do not use bare 'except'

Line 158:9: E722 do not use bare 'except'

Line 137:13: W503 line break before binary operator
Line 144:13: W503 line break before binary operator
Line 512:17: W503 line break before binary operator
Line 516:17: W503 line break before binary operator

Line 150:13: E722 do not use bare 'except'
Line 320:25: W503 line break before binary operator

Line 68:110: E226 missing whitespace around arithmetic operator
Line 146:13: W503 line break before binary operator
Line 147:13: W503 line break before binary operator
Line 148:13: W503 line break before binary operator
Line 148:13: E129 visually indented line with same indent as next logical line
Line 269:13: W503 line break before binary operator
Line 269:13: E129 visually indented line with same indent as next logical line
Line 312:13: E722 do not use bare 'except'
Line 371:16: W503 line break before binary operator
Line 372:16: W503 line break before binary operator
Line 466:20: W503 line break before binary operator
Line 494:13: W503 line break before binary operator
Line 495:13: W503 line break before binary operator
Line 496:9: E124 closing bracket does not match visual indentation

Line 103:13: E722 do not use bare 'except'

Line 179:121: E501 line too long (126 > 120 characters)
Line 180:121: E501 line too long (129 > 120 characters)
Line 181:121: E501 line too long (141 > 120 characters)
Line 182:121: E501 line too long (126 > 120 characters)

Line 51:4: E114 indentation is not a multiple of four (comment)
Line 52:4: E114 indentation is not a multiple of four (comment)
Line 53:4: E114 indentation is not a multiple of four (comment)
Line 72:5: E722 do not use bare 'except'

Line 157:121: E501 line too long (131 > 120 characters)
Line 159:121: E501 line too long (122 > 120 characters)
Line 160:121: E501 line too long (149 > 120 characters)
Line 273:121: E501 line too long (123 > 120 characters)

Line 39:47: E228 missing whitespace around modulo operator

Line 44:44: W503 line break before binary operator
Line 45:44: W503 line break before binary operator
Line 52:44: W503 line break before binary operator
Line 53:44: W503 line break before binary operator
Line 56:44: W503 line break before binary operator
Line 57:44: W503 line break before binary operator

@laggardkernel
Copy link

I don't think this should be handled by ScrapydWeb. Scrapy has a extension scrapy.extensions.closespider.CloseSpider to close spiders on specific condition. Setting CLOSESPIDER_TIMEOUT is what you want.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants