Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More than one scheduler, the error raises about TypeError("'run_spiders' object is not iterable",) #131

Open
itualoha opened this issue Jul 22, 2019 · 0 comments

Comments

@itualoha
Copy link

Firstly, it is an amazing project.

Everything is okay for the example project.

But when I add more than one scheduler for the different kinds of scraper, the background reports the error of type error. The following error information is coming from my test project. How to fix the problem?

[2019-07-21 12:48:19,115: ERROR/MainProcess] Task spideradmin.tasks.run_spiders[59c5b7cc-6078-49b2-8e84-1873edc7479a] raised unexpected: TypeError("'run_spiders' object is not iterable",)
Traceback (most recent call last):
File "/opt/anaconda3/lib/python3.6/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/opt/anaconda3/lib/python3.6/site-packages/celery/app/trace.py", line 438, in protected_call
return self.run(*args, **kwargs)
File "/var/projects/spideradmin/spideradmin/tasks.py", line 18, in run_spiders
t.run_spiders(TendersWebsite, 'scraper', 'scraper_runtime', 'tender', *args, **kwargs)
File "/opt/anaconda3/lib/python3.6/site-packages/dynamic_scraper/utils/task_utils.py", line 73, in run_spiders
ref_obj_list = ref_obj_class.objects.filter(*args, **filter_kwargs).order_by(runtime_field_name + '__next_action_time')[:max]
File "/opt/anaconda3/lib/python3.6/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/opt/anaconda3/lib/python3.6/site-packages/django/db/models/query.py", line 781, in filter
return self._filter_or_exclude(False, *args, **kwargs)
File "/opt/anaconda3/lib/python3.6/site-packages/django/db/models/query.py", line 799, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))
File "/opt/anaconda3/lib/python3.6/site-packages/django/db/models/sql/query.py", line 1260, in add_q
clause, _ = self._add_q(q_object, self.used_aliases)
File "/opt/anaconda3/lib/python3.6/site-packages/django/db/models/sql/query.py", line 1286, in _add_q
allow_joins=allow_joins, split_subq=split_subq,
File "/opt/anaconda3/lib/python3.6/site-packages/django/db/models/sql/query.py", line 1161, in build_filter
arg, value = filter_expr
TypeError: 'run_spiders' object is not iterable

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant