Git Sync improvement

[elitongadotti]

请描述该需求尝试解决的问题
Hello,
I'd like to suggest to improve git sync functionality in order to make it possible for scenarios where there are dozens (or even hundreds) of spiders. Currently the functionality requires that each spiders has its own repository. In such scenarios mentioned before, I would have too much repositories to make it doable.

Currently I'm using as workaround bash script + crontab job to pull data from one single repository on github, where each spider has its own folder, same structure that is found on /app/spiders path.

Best,
Eliton

[tikazyq]

I think this idea is quite making sense and doable. What we can implement is something like "virtual spider", a spider that links to some subdirectories of a git repo.

Welcome to hear your feedback and ideas if you have any suggestion.

[elitongadotti]

Hello.

It is also doable. No idea how complex is following this or that way to make it work, actually. The way I suggested was the way "I did" with bash scripting.

Thanks,
Eliton

[tikazyq]

Perhaps I didn't convey my idea clearly. You can follow this issue and a proposed model/process on file management in Crawlab will be implemented later this or the month after.

[pfrenssen]

There is also a common use case in Scrapy where multiple spiders are hosted in a single folder. This allows spiders to reuse common pipelines and middlewares in a shared code base:

├── my_scraper
│   ├── items.py
│   ├── middlewares.py
│   ├── pipelines
│   │   ├── pipeline1.py
│   │   ├── pipeline2.py
│   │   └── ....
│   ├── settings.py
│   └── spiders
│       ├── spider1.py
│       ├── spider2.py
│       ├── ...

If we have a single codebase with 100 spiders in them, at the moment we need to clone the same repository 100 times, and if there is an update to the repository, we need to pull changes 100x.

It would be really nice if we could point spiders to the same code base, but this would mean to decouple the concept of files / git repos from spiders.