Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

创建项目级数据库,项目下爬虫共享该库。 #1414

Open
glacierck opened this issue Oct 26, 2023 · 0 comments
Open

创建项目级数据库,项目下爬虫共享该库。 #1414

glacierck opened this issue Oct 26, 2023 · 0 comments
Labels
enhancement New feature or request spider Spider related

Comments

@glacierck
Copy link

请描述该需求尝试解决的问题
项目管理目前这是起到了个爬虫的分类管理,在功能上其实啥都没有。
希望同一个项目下的多个爬虫都共享同一个数据库表。具备项目级的夸爬虫的去重功能。
现在我在爬10个新闻站点的新闻,各爬各的最后还得导出数据合并一起合并

请描述您认为可行的解决方案
例如,创建统一的项目级items.py,项目下的爬虫统统都得继承这个类,这样就能统一了项目下的多个爬虫的输出。

@glacierck glacierck added the enhancement New feature or request label Oct 26, 2023
@tikazyq tikazyq added the spider Spider related label Oct 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request spider Spider related
Projects
None yet
Development

No branches or pull requests

2 participants