-
Notifications
You must be signed in to change notification settings - Fork 324
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TypeError: can't pickle thread.lock objects #252
Comments
Your error seems to be happening in this method - perhaps you are adding a non-serializable field to your request? you can disable that logging middleware or simply change what it is logging to help you understand what is happening. |
Thanks for the help, I will give it a shot |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi.
I don't know how much it happens or how much it already happenned but one of my crawl fell on the error below.
I ran a thousand requests and it happened to only one of them. But still my crawl was put down. Here is the stack trace:
2021-02-08 15:03:30 [scrapy.core.scraper] ERROR: Error downloading <GET https://pccomponentes-prod.mirakl.net/login>
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
result = g.send(result)
File "/usr/local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 66, in process_exception
spider=spider)
File "/usr/src/app/crawler/crawling/log_retry_middleware.py", line 89, in process_exception
self._log_retry(request, exception, spider)
File "/usr/src/app/crawler/crawling/log_retry_middleware.py", line 102, in _log_retry
self.logger.error('Scraper Retry', extra=extras)
File "/usr/src/app/crawler/scutils/log_factory.py", line 244, in error
extras = self.add_extras(extra, "ERROR")
File "/usr/src/app/crawler/scutils/log_factory.py", line 319, in add_extras
my_copy = copy.deepcopy(dict)
File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/usr/local/lib/python2.7/copy.py", line 264, in _deepcopy_method
return type(x)(x.im_func, deepcopy(x.im_self, memo), x.im_class)
File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/usr/local/lib/python2.7/copy.py", line 230, in _deepcopy_list
y.append(deepcopy(a, memo))
File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/lib/python2.7/copy.py", line 182, in deepcopy
rv = reductor(2)
TypeError: can't pickle thread.lock objects
Some help would be highly appreciated
The text was updated successfully, but these errors were encountered: